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Foreword 


Theoretically, Politics of Quality in Education is a very solid and consistent book. 
Its authors bring new elements to an understanding of one of the central themes 
in the configuration of education policies in recent decades. 

Based on a comparative study of three BRICS countries — Brazil, China, and 
Russia — new perspectives are opened for thinking about and interpreting “how 
the emphasis on quality has changed the basic conditions in which education 
takes place” (this book, p. 1). 

As David Labaree states, in the twenty-first century, the education policy 
discourse converges on a single overarching goal for education: 


From the global education reform movement to its policy apparatus in the 
OECD and its policy police in the PISA testing program, we have seen 
one goal trump the others ... The OECD approach serves to narrow the 
learning outcomes of schooling and educational knowledge production to 
whatever is in service to economic development. 


(2017: pp. 281-282) 


The argument of the book is developed against this background. The idea of qual- 
ity in education, as it is circulated throughout the world, is here operationalised as 
a governance tool. It is clear from the outset that the central issues “for analysing 
quality in education globally are how quality is connected to transnational flows 
of knowledge and how it offers a powerful governance tool” (this book, p. 5). 

The authors reveal an excellent knowledge of the state of the art but above 
all a rare intellectual sophistication. They do not fall into pre-established theo- 
retical frameworks which explain everything even before the research is under- 
taken or into dichotomies which so often close and diminish the interpretation 
of educational and political issues. 

On the contrary, throughout the book, we are driven to a systematic prob- 
lematisation, opening new interpretive paths and new perspectives to consider 
the political themes of quality in education. Written with elegance and sagacity, 
the book does not seek to avoid complexity. There are no easy answers but rather 
an invitation for the reader to subscribe to multiple interwoven interpretations. 
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The authors explain, “Before it is operationally defined, ‘quality in education’ 
remains abstract and elusive ... it is not until it is put into practice, or “opera- 
tionalised’, that it becomes definable” (this book, p. 1). In this sense, they seek to 
investigate how the operationalisation of quality takes place and the repercus- 
sions this has on the room for action of the different actors involved in education. 

They assume the need to take into account, simultaneously, the world’s com- 
plexity and contingency to understand the political processes in the education 
arena. Instead of discussing questions about source and target, global and local, 
convergence and divergence, they are interested in understanding networks, 
flows, and complex causation. 

The research is organised around a triangle whose vertices are the political 
situation, the political possibilities, and the use of political space. Through this 
scheme, the authors seek to understand the politics of quality in Brazil, China, 
and Russia: first, they explain how quality assurance and evaluation strategies 
have become the goal of education instead of quality itself; then, they describe 
how these strategies “[enable] a parallel trend of authorising more governance 
methods, but at the same time [create] increasingly complex systems” (this book, 
p. 11); finally, they clarify how these strategies bring new actors and reconfigura- 
tions of the field and point to their potential for changing policies in education. 

In this foreword, I would like to underline two points that are relevant for an 
understanding of the intellectual effort made in this work. 

The first concerns the need to go beyond the traditional divisions of space, 
either in the opposition between the global and the local or in a strict vision of 
national boundaries, which typically confine comparative studies to rigid and 
uninteresting frameworks. 

Nowadays, the question does not lie mainly in a logic of transfer, export, 
and import, but rather in understanding how we all, through our own network 
presence and collaboration, contribute first to the delocalisation and then to the 
relocalisation of policies. It is not about a space with rigid locations or without 
locations but instead about a space with multiple locations. 

The second point concerns the attention given in recent years to the dema- 
terialisation of space, with a more complex analysis of flows, communications, 
networks, and connections. These studies often tend to accentuate fluidity as if it 
were fluctuation, without paying due attention to key dimensions brought to the 
scientific debate by authors like Anna Tsing or Tim Cresswell. I am referring 
to the concepts of friction and turbulence. 

Anna Tsing (2005) explains that without friction, movement would be 
impossible. The surfaces would slide into each other in a frictionless fluidity, 
which would make any kind of traction — that is, movement — impossible. 

Likewise, Tim Cresswell (2010), working on mobilities, invites us not to 
see turbulence as a moment of breakdown — a fundamentally bad thing — but 
rather as an event of creative potentiality. Moments of turbulence produce sud- 
den visibilities in systems which would otherwise remain mostly invisible and 
taken for granted. 
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These two perspectives cross the book, edited by Jaakko Kauko, Risto Rinne, 
and Tuomas Takala, in which the ideas of complexity and contingency are 
taken as an epistemological starting point. 

The major interest of this research is its ability to advance interpretations 
which escape the traditional frameworks and allow us to understand the impor- 
tance of the politics of quality in education in three different countries. The 
interpretative potentialities of a research which crosses several countries, in this 
case Brazil, China, and Russia, become evident in this book, which brings an 
important renewal to comparative studies. 


XX 


Methodologically, this work is very thoughtful and innovative, opening new 
and important perspectives for comparative studies in education. I would like 
to draw attention to two dimensions which are very relevant in the way the 
research was conducted. 

The first is the selection of the countries on which the comparison is based. 
The idea of choosing them from the BRICS universe is particularly interest- 
ing, because we find here a political intention to bring together very different 
countries, with very different histories, but all of them experiencing a historical 
period of affirmation on the international scene. 

This choice allows one of the central research questions to be posed: can we 
consider that these countries are in a transition phase from “norm-takers” to 
“norm-makers” in the global environment? 

The conclusions refer to three different ways of approaching the relation- 
ship with the politics of quality in education: in Brazil, a strong pace towards 
assessment and a slow pace towards quality are identified; in China, the imperial 
legacies and the needs of the party-state, leading to the construction of QAE 
strategies with Chinese characteristics, are mentioned; finally, it is suggested that 
in Russia, quality control serves to reinforce central regulation. 

In general, we can consider that global policies of quality in education prevail 
over any alternatives tried in the BRICS space. It is true that, under the New 
Delhi Declaration of Education, signed in 2016, these countries commit them- 
selves to “share the best practices available in BRICS countries in collaboration 
in education, research and innovation through the BRICS Network Univer- 
sity”. However, one must recognise the political fragility of these intentions. 

It is in this sense that one can interpret the occasionally harsh judgements 
which heads of international organisations, for example PISA, allow themselves 
to address to national officials who fail to follow their “recommendations”. 

In this respect, an interview with Andreas Schleicher, Education Policy 
Officer of the OECD, with Veja, a leading Brazilian magazine, is revealing: 


Brazil has begun to have a chance to move forward from the moment 
that it began to map problems objectively — and no longer on the basis 
of the intuition of the rulers. This is basic. We cannot think of improving 
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what has not even been measured. Hence the importance of international 
comparison. 


(see Weinberg 2008: pp. 20-21) 


The central goal of PISA, well beyond the BRICS, is to contribute to the 
“modernisation” of education systems in the face of the challenges of the 
knowledge economy and globalisation. 

In other words, it is a matter of defining what each country must “neces- 
sarily” embrace and implement. Each state in its “freedom” must “inevitably” 
adopt these recommendations. Indeed, indicators, and mainly quality indica- 
tors, are never a mere description of reality. They provide a way of constructing 
reality, of defining what is important and what is not, of establishing priorities, 
which inevitably influence actions and policies. And the situation is no different 
in the world of BRICS. 

The second dimension which seems relevant to me is the choice of compar- 
ative research strategies. The authors do not confine themselves to traditional 
methodologies, whether country-to-country comparisons or macro-analyses as 
developed by world-system approaches. 

They are able to define intermediate levels of comparison which do not 
ignore the historical specificity of each country or the interest of a more general 
view but which allow them to build a more sophisticated, strongly comparative, 
open, and problematising analysis. They call their approach “comparing dynam- 
ics”, emphasising that they are interested in movement in education politics. 

This is what allows them to elucidate not so much cross-national compari- 
sons but the creation and ongoing re-creations of “global signifiers” based on 
international competition and assessments. This process leads to the diffusion 
of global patterns and flows of knowledge which are assumed to be applicable 
in various places. These international indicators and benchmarks are not spon- 
taneously generated. On the contrary, they are the result of policy-oriented 
educational and social research. 

The work of methodological construction carried out in this project, both in 
the initial phase of conception and in the development of the cross-analysis of 
the three cases, deserves special mention. We are facing significant advances in 
comparative work in education, which means that this book will be mandatory 
reading for researchers wishing to embark on comparative studies in the field 
of education and politics. 

In addition to these two dimensions, it is fair to point out the authors’ care 
in the organisation of their research, as well as in the analysis and interpretation 
of the data. It is very interesting that they assume, from the outset, that “the key 
to maintaining validity in a qualitative research project such as ours is to adopt 
a reflective approach throughout” (this book, p. 18). 

Indeed, throughout the book there is an attitude of self-reflexivity, which 
leads the authors to relativise their advanced interpretations. They explain this 
permanent exercise through a research which is conscious of the interpretation 
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made, prudent with the systematics of the research procedure, aware of the 
political and ideological role of social science research, and reflective in terms 
of representation and authority. 

This attitude leads them to a permanent swing between the explanation 
of their interpretations and the cultivation of a doubt, which opens space for 
other interpretations. We all know that the best science feeds on doubt and 
does not fall into the arrogance of certainties, which are the shortest route to 
ignorance. 


XX 


Intellectually, this is a bright book. The authors produce a very important set 
of reflections and open important perspectives for future studies, in particular 
concerning the role of experts and quality policies in education. In this sense, 
they explain that changes in the political agenda of the countries analysed are 
related to changes in actor constellations. 

The analysis of national and international experts and the role they play is 
crucial for the argument developed in the book. The way they influence deci- 
sions and change the political landscape is particularly relevant in the debate on 
quality education issues. 

In this regard, I would like to return to the criticism made forty years ago by 
a very influential, but easily forgotten, author, Ivan Illich: 


Experts and an expert culture always call for more experts. Experts also have 
a tendency to cartelize themselves by creating “institutional barricades” — for 
example proclaiming themselves gatekeepers, as well as self-selecting them- 
selves. Finally, experts control knowledge production, as they decide what 
valid and legitimate knowledge is, and how its acquisition is sanctioned. 
(see Finger & Astin 2001: p. 10) 


As this book explains, education policies are increasingly influenced by an 
expert-discourse. However, after so many studies confirming this over-presence, 
it is worth returning to a concept developed by Ivan Illich: counterproductivity. 

Counterproductivity is the means by which an intrinsically useful and ben- 
eficial process can turn into something negative. Ilich’s idea applies to different 
situations and contexts: medicalisation of health, schooling, car traffic, technolo- 
gies, and so on. He argues that beyond a certain point medicine generates dis- 
ease, transportation generates congestion, and education causes adverse effects 
(lich 1977: p. 28). 

I believe this critique remains accurate. It can easily be applied to the produc- 
tion of statistical indicators which, beyond a certain point, proves counterpro- 
ductive. In this sense, we need to imagine a different, more critical and reflective 
use of the “numbers” available to us. 

Indicators are an important support for our interpretations of the world, 
but they cannot take the place of our judgements and choices. Comparison is 
a way of understanding the world critically but not a way of governing it, as if 
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policies were a “simple”, “neutral”, and “objective” administration of the results 
presented in a list of indicators. 

In this sense, comparative work in education should not serve to “expro- 
priate” citizens” decision-making capacity in favour of experts and an expert- 
discourse but rather to provide elements and interpretations which allow them 
to “appropriate” politics. 

We have an urgent need to proceed with a methodological revolution of 
comparative approaches. It is no longer possible to gaze fixedly at the concrete 
ground of “realities” which do not allow comparative reflection. It is no longer 
possible to generalise concepts without roots by constructing a vague and float- 
ing thought. 

Like history, comparative research must not focus on “facts” or “realities” but 
on problems. The “facts” — events, countries, systems, and so on — are, by defini- 
tion, incomparable. We can illuminate the “specificities” and “similarities”, but 
we cannot go further. Only problems can be the basis of a comparative reflec- 
tion, because they situate us in the face of our memories and imaginations, 
producing new areas of examining a space which is not delimited by physical 
boundaries but rather by boundaries of meaning. 

For a long time, democratisation was at the forefront of education research. 
We must re-establish this lost tradition, against the myth of the “objective 
expert”. It is necessary to rebuild a culture of discussion and debate, of reading 
each others work and engaging in meaningful intellectual conversation. We 
must reinvent research as an open collective praxis. 

Our best tradition relies on multiple approaches and ways of thinking. We 
need to enrich, to deepen, and to diversify our understanding of education 
matters, which also happens through comparison. There is no simple way to 
accomplish this goal. We cannot hope to reach a consensus on how to organise 
and conduct research in education. But we can engage in an intellectual dia- 
logue to achieve convergence and trust. Nothing will be achieved without time, 
collaboration, and collective commitment. 

Furthermore, it is crucial to open research to a wide public. We need to “go 
public”, to connect our research with public purposes and expectations. To 
conduct research — and especially comparative research in education — requires 
an effort of imagination and logic and demands the production of a scientific 
perspective which we can communicate to others. Lucidity is what we need 
from education comparatists — a lucidity evident in this book, a book which 
deserves to be read, discussed, and continued. 

Antonio Novoa 
University of Lisbon 
15th December 2017 
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1 Comparing politics of quality 
in education 


Jaakko Kauko, Tuomas Takala, and Risto Rinne 


Introduction 


This book is about understanding the politics of quality in education. Under- 
standing this politics is our goal because the question of quality has become 
one of the most important framing factors in education policy and practice. It 
has been of growing interest to international organisations and national poli- 
cies since at least the 1990s (Leeuw 2002; Power 1994; Smith 1990). We are 
interested in how the emphasis on quality has changed the basic conditions in 
which education takes place. 

Before it is operationally defined, “quality in education” remains abstract and 
elusive. Like “choice” or “standards”, it may have rhetorical appeal as a gener- 
ally desirable idea, but it is not until it is put into practice, or “operationalised”, 
that it becomes definable (Kauko et al. 2016; Dahler-Larsen 2015). The same is 
true for the evaluation of quality. There is little consensus currently even among 
evaluators as to what constitutes sound evaluation, with some adopting nearly 
opposite methodological and theoretical stances. (Dahler-Larsen 2012; Grek 
et al. 2009; Karlsson Vestman & Segerholm 2009). 

Our research seeks to investigate how quality is operationalised and the 
repercussions this has for the room for action for the different actors involved 
in education. By actors, we mean the teachers, students, parents, politicians, and 
civil servants and the organisations they represent who have different room for 
action, from the local to the global. 

We focus on what we call quality assurance and evaluation, QAE (see Ozga 
et al. 2011). “Quality assurance” implies the need to demonstrate quality (of 
education) to outsiders (Harvey 2004—17); “evaluation” refers to the general 
aim of learning and changing current practices, providing “retrospective assess- 
ment of public-sector interventions, their organisation, content, implementa- 
tion and outputs or outcomes, which is intended to play a role in future practical 
situations” (Vedung 2010). Our definition of QAE is intentionally wide, and 
it encompasses an array of activities used to evaluate and assure quality, such as 
student assessment and testing, programme and policy evaluation, school per- 
sonnel and institutional evaluation, accreditation, curriculum evaluation, self- 
evaluation, inspection, and auditing (Kellaghan, Stufflebeam, & Wingate 2003). 
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The most common instance of QAE under scrutiny in this book is represented 
by national large-scale assessments of student achievement. 

The importance of understanding QAE is that it is a tool to exercise power 
(Dahler-Larsen 2015: pp. 29-31), regardless of whether we take a classical view 
of power as related to resources, attributes, and formal positions (Mills 1956; 
Dahl 1961;see Bachrach & Baratz 1962; Lukes 1974) or see it as more relational 
and structural (e.g., Foucault 2003; Heiskala 2001). Indeed, QAE is a govern- 
ance tool, which means that procedures intended to enhance quality in educa- 
tion with the help of evaluation are used to achieve political aims, and they 
shape the power situation of the actors involved. 

If one accepts our definition of QAE as a tool of power, it is no surprise that 
QAE has been embraced in policymaking. For example, international organisa- 
tions and national governments are greatly optimistic in their use of QAE as 
a tool of governance. In the field of comparative education, there is vast and 
growing research into the international and transnational formation of educa- 
tion policy agendas and their influence on policymaking in individual coun- 
tries (e.g., Chabbott 2002; Mundy 2007; Steiner-Khamsi and Waldow 2012; 
Verger, Novelli, & Altinyelken 2012). The concepts of the international and 
the transnational are even sometimes used interchangeably, but by the former, 
we mean the actions which happen between national actors or with the help 
of international organisations, while by the latter, we mean the unofficial and 
official networks of action and ideas which exist without a restrictive relation- 
ship with national borders. Much of the literature mentioned focuses on how 
international organisations may guide or condition national education policy 
decisions. They do this by framing policy discussions and providing related 
technical advice and by prioritising the promise of external funding for projects 
and sector programmes (e.g., Beech 2006; King 1991). 

Rational planning with the help of QAE contrasts with the long tradition 
of research literature claiming that policymaking never has the grassroots result 
politicians expect. Studies of the policy process (e.g., Jenkins-Smith & Sabatier 
1993; Baumgartner & Jones 2009) have pointed out the institutional limitations 
created by history (Pierson 2000) or norms (March & Olsen 1989) and drawn on 
chaotic-sounding notions, such as organised anarchy or “garbage can” (Cohen, 
March, & Olsen 1972; Kingdon 2003). In education, there are also many gener- 
alising analyses of how reforms fail (e.g., Ball 2001). As Tyack and Cuban (1995) 
note, it is important to understand how schools act to change intended reforms. 

This book is a result of the research project “Transnational Dynamics of 
Quality Assurance and Evaluation Policies in Brazil, China, and Russia”, 
funded by the Academy of Finland between 2014 and 2017 (grant numbers 
307310, 273871, 274218, 273874). During these four years, we conducted 200 
interviews with 278 people from different levels of action, conducted observa- 
tions, and analysed documents. The work was undertaken in three sub-projects, 
which were responsible for data collection in each country. Crucially, research- 
ers were fluent in the respective national languages of our case countries. 
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The challenge of policy planning is linked to the book’s ontological assumptions — 
the complexity and contingency of the world. We understand them as coher- 
ent and overlapping descriptions of the same thing from the perspectives of 
complexity theory (complexity) and social sciences (contingency). Complex- 
ity theories share ideas of systemic behaviour through the interaction of its 
parts (e.g., emergence and positive and negative feedback), non-linear causa- 
tion (e.g., partial determination, probabilistic event progression), and different 
kinds of branching effects (e.g., path dependencies, irreversibility, or punctuated 
equilibria) (Wimmer 2006; Cairney & Geyer 2015; Kauko 2014; Room 2015; 
Biesta & Osberg 2010). In the social sciences, contingency means that change 
trajectories are possible but not necessary (Joas 2004; Medd 2002; see Kauko & 
Wermke 2018). Politics is thus essentially the organisation and reorganisation 
of contingency (Kauko 2014; Palonen 1993, 2003, 2006): it is a game in which 
the basic conditions and constraints for action are defined. 

QAE is a good example of the attempt to take political control of complex- 
ity. These attempts to channel and control contingency take different shapes. 
Classically, in political science and complexity theory, a basic limitation is pre- 
sented by institutions and their formation: 


Institutions constrain and channel agent interactions; however, they can 
also be subverted from below or reformed from above, as actors lift their 
gaze, reflect upon the overall socio-economic system in which they live 
and reinforce or reshape the rules and architectures of those systems. This is 
the stuff of politics and political choice ... It accepts the potential value of 
modelling social dynamics as a self-organizing system, analogous to those 
in the natural sciences. On the other hand, it insists that social science must 
also be centrally interested in the socio-political processes by which these 
dynamics are re-shaped. 

(Room 2015: p. 20) 


We attempt to build an understanding of these socio-political processes and 
their enabling or restriction of the dynamics possible in the discussed setting. 
This also requires an understanding of history in observing path dependencies 
(Capano 2009). They indicate how past solutions create a dynamic of increas- 
ing returns (Pierson 2000) in which the costs of reversal due to institutional 
reorganisations grow continually (Levi 1997). These events in history shape the 
room for political action. 

In short, our premises seek to understand the contingent and almost fragile 
context of political action and how it is always tied to longer-term socio- 
historical developments, constrained or enabled by institutional rearrange- 
ments, and how actors are both entangled in and empowered by the dynamics 
which develop in these conditions. 
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We have discussed how the ideas of complexity and contingency provide a 
good basis for not expecting an ideal of a conscious implementation process 
and have also questioned the ease with which QAE can be used as a govern- 
ance tool. So, how is it possible to study this phenomenon? 


Unpacking policy transfer 


In terms of disciplinary traditions and boundaries, our book could be placed in 
the fields of comparative education, political science, or even international rela- 
tions. In some cases, there may be more variance within a discipline than between 
disciplines. However, our main discussion is largely concerned with the field of 
comparative education, which influences our discussion across disciplines. 

We agree with Rappleye (2012), who argues for a complex understanding of 
policy transfer and that in a globalising world, the question is crucial. He criti- 
cally examines the prominent debate which circulates around a large body of 
system theories like the borrowing and lending theories, which especially ana- 
lyse the influence of globalisation from the local perspective, and world culture 
theories, which see convergence in the structures of education systems glob- 
ally (Rappleye 2012). Borrowing and lending theories study policy diffusion 
(Steiner-Khamsi 2004: p. 213), ranging from cross-cultural attraction to policy 
tourism (Steiner-Khamsi 2012). The leading idea of world culture theory is 
that educational systems around the world are developing in a similar direction 
structurally mainly as a result of communication amongst international organi- 
sations as they diffuse policy blueprints (Meyer, Bol, Thomas & Ramirez 1997; 
Ramirez 2012). Disagreement between these views relates to the epistemologi- 
cal understanding of change (Kauko & Wermke 2018) and the degrees of local 
and global influence (Rappleye 2012). According to Rappleye (2012: p. 49), the 
entrenched debate between these two perspectives has led to a “theoretical and 
conceptual fatigue”. 

Our research is interested in the effects of what is generally called policy 
transfer, but we would like to be more specific concerning the causality this 
implies for these traditions. Waldow (2012) sums up borrowing and lending 
theories as sharing the ideas of standardisation and legitimation. Standardisation 
refers to intentional and unintentional movement in a common direction and 
production of legitimacy to how external references are used to make internal 
changes or how policies are borrowed from a reference society. World culture 
theories do not, at least explicitly, consider reference societies but instead look 
at how nations influenced by world culture start consensually changing their 
systems towards a common structure (Meyer & Ramirez 2003). Put simply, 
in both cases, diffusion bears the idea of a policy moving from one place to 
another, connoting three things: there must be a “source” (a reference society, 
international organisation, or “world culture”), there needs to be a recipient, 
and there needs to be an observable policy (or a “blueprint”). 

Our premise differs from these usual views of policy transfer. The analytical 
framework we endorse in this book leads to a disagreement with the notion 
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of policy transfer, which in our view cannot cope with a sufficient degree of 
complexity for a thorough analysis of the process. Instead of discussing questions 
regarding source and target, global and local, and convergence and divergence, 
we believe that source and recipient become blurred in networks and flows and 
understanding the effects of policy transfer, embedded as it is in complex causa- 
tion, requires theoretical tools to deal with contingencies, path dependencies, and 
probabilities. Ideas like these have been recognised, for example, under the label 
of post-structuralist theories (e.g., Larsen & Beech 2014; Carney 2009) or when 
Schriewer (2003: p. 20) seeks to understand the “complexity of causal networks”. 

In analysing interactions between actors, we arrive at an understanding of 
dynamics as contingent. In the context of policy transfer, complex causation 
refocuses our analysis from the attempt to understand a policy’s source, recipi- 
ent, and enactment to an attempt to understand the web of factors which 
exist in a socio-historical situation and sustain a certain dynamic of change. 
We argue that central questions for analysing quality in education globally are 
1) how quality is connected to transnational flows of knowledge and 2) how 
it offers a powerful governance tool for shaping education for several actors, 
including for some we would not usually consider. 


The transnational agenda of QAE 


Given our starting points, QAE in education offers a fruitful point for analysis, 
because it brings to the fore the theoretical aspects in which we are interested. 
QAE is a gateway to understanding transnational flows, because international 
and national actors use it as a tool of governance: measurement in education 
facilitates the creation of categories for the performance of individuals or 
organisations and for making decisions (Fenwick, Mangez, & Ozga 2014). 

The increased role of knowledge and data management in the making of edu- 
cation policy has been analysed as a recent development in the modes of gov- 
ernance (Carvalho 2013; Fenwick, Mangez, & Ozga 2014; Lawn & Segerholm 
2011). As part of this change policy, advice has increasingly conveyed messages 
of “international best practice”. Organisations such as the World Bank and the 
Organisation for Economic Cooperation and Development (OECD) have had 
an important role in the dissemination of these policies. These international 
organisations can be characterised as “epistemic communities” drawing on the 
knowledge dependency created by the increasing requirements for decision- 
makers to deal with a broader range of issues in the expanding global economy 
and the consequent need for external advice (Armingeon & Beyeler 2004; Haas 
1992; Hasenclever, Mayer, & Rittberger 2002; Kallo 2009: p. 357). 

Our analysis of QAE is restricted to general school education (i.e., it excludes 
vocational and tertiary education). The transnational agenda around QAE is 
constructed within a network of actors which have no clear boundaries or 
identifiable centre of power or thought from which the agenda emanates. 
The network shares a common belief in the necessity of measuring learning 
outcomes as a key precondition for improving education quality worldwide 
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(e.g., UNESCO Institute of Statistics 2013: pp. 14-16; World Bank 2013: p. 4; 
Center for Global Development 2013). Actors within the network consist of 
a wide range of multilateral organisations, for example, the World Bank; the 
OECD; the United Nations Educational, Scientific and Cultural Organisation 
(UNESCO); regional networks, such as the Latin American Laboratory for the 
Assessment of Quality in Education (LLECE) and the Southern African Con- 
sortium for the Measurement of Education Quality (SACMEQ); private profes- 
sional bodies (the Brookings Institute, Educational Testing Services); academic 
experts; and consultants. 

The policies of the three principal education-related international organisa- 
tions relevant to QAE — the World Bank, UNESCO, and the OECD — vary in 
how their assistance is tied to money flows and the extent to which national and 
local actors are able to set the agenda. Whereas the World Bank is perceived as 
leaning towards a harder policy line, with leverage provided by its lending opera- 
tions, the OECD has authority without financial clout, and UNESCO has been 
reduced to a relatively weak policy actor (Henry et al. 2001: pp. 17-18; Rinne & 
Ozga 2011). While these organisations have different agendas, especially the 
World Bank and the OECD, and to an extent UNESCO, they share a similar 
toolkit:“a range of instruments that can be targeted at national policies: they can 
promote, develop and disseminate policies, coordinate, set standards, supply tech- 
nical assistance and offer financial inducements” (Rinne & Ozga 2013: p. 98). 

The improvement of education quality can of course be justified by refer- 
ence to any objective set for the education system. In contrast, the typical jus- 
tification expressed for the transnational assessment agenda is notably narrow 
and even monotonous: reference is made in numerous sources to the finding 
that cognitive learning achievement — rather than years of schooling per se — has 
a positive impact on economic growth (original research paper by Hanushek & 
Woessmann 2007; references, e.g., World Bank 2011: p. 24; Center for Global 
Development 2013: pp. 3-4). The corollary is that a well-functioning system of 
assessing learning outcomes is “a key driver of economic growth and poverty 
reduction” (READ 2010: p. 36). 

The “problematisation” and overall policy advice on QAE in general school 
education found in the global World Bank documents in the past two decades 
illustrates the substance of the transnational agenda: 


1) The principal goal of education is seen as contributing to economic growth 
and poverty reduction. 

2) This contribution is crucially dependent on the quality of education. 

3) There is mounting evidence from international assessments that the quality 
of education is low in many countries. 

4) Information on learning achievement is needed both for monitoring edu- 
cation quality as a basis for policy decisions and for increased accountability 
of schools towards parents, communities, and donor agencies; such infor- 
mation should then provide incentives for improving quality. 

5) The case is then made for the establishment of national assessment systems 
and wider participation in international assessments and related capacity 
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building; to these, the World Bank and other agencies should provide sup- 
port (World Bank 1995: pp. 1-3, 9, 15; 1999: pp. iii, vii-ix; 2005: pp. 12, 
67-69; 2011: pp. 6-8). 


The role of the OECD derives from its knowledge-production capacity. It is 
thought to have a central role in the flow of international educational ideas and 
in the governance of education (Rinne & Ozga 2011), and the market-liberal 
ideological undercurrents in knowledge-making have also been noted (Dale 
1999: pp. 1-4; Dale & Robertson 2002: p. 11). The OECD’s central role in set- 
ting international benchmarks makes it a broker of the main currency in the 
global education setting, which national policymakers usually see as a global 
competition (Lauder et al. 2006: p. 41; Robertson, Bonal Dale. 2002; Rinne, 
Kallo & Hokka. 2004; Rinne & Ozga 2011). The OECD' annual compen- 
dium, “Education at a Glance”, has become a statistical “doxa”, just as the PISA 
survey (Programme for International Student Assessment) and its indicators, 
rankings, and league tables are often taken as objective indicators which point 
the way to the improvement of results and placings in rankings in national edu- 
cational policy (Rinne et al. 2004; Rinne & Ozga 2011). Initially confined to 
OECD member countries, the PISA programme has been gradually expanded 
to non-member countries — including Brazil, China, and Russia — which share 
a history of having been clients of the World Bank. For such countries, this 
represents a significant symbolic upgrading of their international status. 

The research on global agendas in education is nuanced and complex and 
deals with many of the basic problematics in the comparative education field 
(Verger, Novelli, & Altinyelken 2012), which tend to emphasise the importance 
of international and transnational changes and their reshaping of the role of 
the state (Werner & Zimmermann 2006; Robertson, Bonal, & Dale 2002). 
Structurally, research seems to indicate the state's diminishing role as an educa- 
tion actor in the face of globalisation. The state is, on one hand, riddled with 
transnational influences, while, on the other, the global QAE agenda allows 
less independence. Where action is concerned, the content of the transnational 
QAE agenda seems to ignore the political undercurrents of QAE — the nego- 
tiations, interpretations, and difficulties highlighted in the above-mentioned 
research tradition concerning policymaking and governance. Using our analyti- 
cal framework, we would expect to find more contingency and room for action 
in how transnational QAE data are used. We also analyse Russia and China, in 
which the state is heavily involved, thus critically examining the discussion of 
the state's diminishing importance. 


Comparing dynamics in education politics 
in Brazil, China, and Russia 


The case countries in our research are Brazil, China, and Russia. Their histori- 
cal trajectories are described in more detail in Chapter 3. Here it is sufficient 
to mention that they present a clear set of common features which justify 
their selection for a comparative study. The term BRICS, coined in the new 
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millennium (O’Neill 2001; Hurrel 2006) and applied to these three “emerging 
economies”, along with India and South Africa, has been adopted by the case 
countries themselves, suggesting their global ambitions are somewhat similar. 
Brazil, China, and Russia aspire to regional leadership and leverage their eco- 
nomic and political positions through social policies, including education. Like 
other countries, they have increasingly focused on QAE in education (Chin 
2012; Hurrel 2006; Lima 2012; Piattoeva & Takala 2014; Rowlands 2012). In 
these three contexts, we analyse how QAE works in transnational networks, 
how data circulate, how QAE functions as a governance tool, and how all this 
affects the room for action available to different actors. 

The terms “framework”, “theory”, and “model” are sometimes used inter- 
changeably, but we follow Elinor Ostrom (2005: pp. 27-28), who understands 
them as nested. With the help of a framework, it is possible to loosely identify 
the essential elements for analysis: “They attempt to identify the universal ele- 
ments that any relevant theory would need to include .. . [Elements con- 
tained in a framework help the analyst generate the questions that need to be 
addressed when first conducting an analysis” (Ostrom 2005: p. 28). Theory 
helps to focus the view further in making more specific assumptions: “a scholar 
needs to select one or more theories to use in generating predictions about 
expected patterns of relationships” (Ostrom 2005: p. 28). Models make more 
precise assumptions concerning a theory's sub-section. However, this is beyond 
the focus of our research. To understand QAE in education in the transnational 
context, we start our analysis with an analytical framework called Comparative 
Analytics of Dynamics in Education Politics (CADEP) (see Kauko et al. 2015; 
Simola et al. 2017). This idea is further refined with compatible theories in the 
subsequent chapters. 

Using CADEP, we hope to tease out comparable and related patterns of 
action. Our framework’s first word, “comparative”, indicates that we analyse 
homologies between the different contexts. The second word, “analytics”, indi- 
cates that we are interested in understanding actors’ room for action. By focus- 
ing on “dynamics in education politics”, we stress the importance of analysing 
the actual movement of the education system rather than focusing on actors’ 
attributes. We thus shed light on the limits and possibilities for action on dif- 
ferent levels. Simola et al. (2017) describe the dynamics in education politics 
as follows: 


In many cases it [dynamics in education] is a result of contingencies in his- 
tory, and is currently sustained by political action on different levels or that 
is constantly subjected to transnational flows. The action itself, regardless 
of whether or not it is considered political, derives from societal thought 
structures, is questioned or unquestioned, happens in the course of time, is 
connected to resources and past events, passes, and creates room for future 
action. What is called equality, trust, or progress is manifested through 
action, described here in terms of relational and contingent dynamics. 
(Simola et al. 2017) 
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To sum up, in attempting to grasp the multifaceted nature of both QAE and 
transnational flows, we share many of the ontological and epistemological 
premises of complexity theories. We emphasise the contingent nature of the 
world, the fact that events and change are possible but not necessary. Described 
in more detail in the next chapter, the three dimensions of the CADEP frame- 
work are derived from a vast conceptual-historical project (Palonen 2003, 
2006) and analysis of the policy process (Kingdon 2003; Baumgartner & Jones 
2009; Sabatier & Jenkins-Smith 1993). 


e Political situation relates to the changing political constellations of actors associ- 
ated with education quality and their effect on what is structurally possible 
on different scales of action. 

e Analysis of political possibilities happens through the dominant discourses: what 
the relevant actors consider possible or what they consider to be the problem. 

e The third dimension is the use of the political space. It relates to how the 
actors use the room for action created by the first two dimensions. 


With the help of these three dimensions, our aim is to analyse how QAE as a 
transnational flow and governance tool changes the role of state, expertise, and 
governance and the room for action which exists for politicians, teachers, and 
other actors in varying contexts. 


The structure of the book 


Chapter 2 discusses the theoretical and analytical premises of our research and 
aims to increase its validity through methodological transparency. It draws 
on the idea of reflective research and discusses the questions of case selection 
and comparability, as well as the concrete work of data collection and analysis. 
Research journey narratives are used to consider field access and ethical issues 
and to produce another layer of research data about the manifestations of QAE. 
The chapter also reflects on the differences in data availability in the three 
countries and the limitations this poses for comparative analysis. To compare 
the dynamics which emerge in QAE policies in the different contexts, the 
chapter introduces the CADEP analytical framework in more depth and dis- 
cusses the abductive research design, in which theoretical and empirical work 
are complementary. 

Chapter 3 analyses the socio-historical background necessary for an under- 
standing of the development of QAE policies in the three countries and for an 
analysis of the problematics in the following chapters. It outlines the historical 
paths of the development of QAE policies in each context during the post- 
World War II period. First, it presents a case-by-case chronological account 
of these trajectories, based on document analysis and literature review; then it 
proceeds to a comparative analysis. 

Chapter 4 is the first to use data from our fieldwork. Following the analyti- 
cal framework, it analyses the changing national political arena in the Brazilian, 
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Chinese, and Russian contexts. In this chapter, we ask how QAE has changed 
the roles of national and international actors in policymaking. We analyse 
interviews with representatives of international and national organisations and 
with individual actors and use document analysis as background material. We 
demonstrate that enhanced data infrastructure and expertise, along with other 
resources, have made national governments less dependent on international 
organisations, while remaining more interested in and interlinked with inter- 
national trends. 

Chapter 5 describes and analyses the changing roles of and relationship 
between the state and expertise in Brazil, China, and Russia. National policies 
aim to determine the development and application of QAE instruments as 
they empower state governance. However, the data generated are also strongly 
dependent on experts. In analysing interview data from experts, civil servants, 
politicians, and other stakeholders, we aim to discover who is considered a 
legitimate collector and analyst of QAE data. We find that a growing number 
of state-controlled systems allow experts some room for action but that state 
approval continues to serve as the main source of legitimation. 

Chapter 6 discusses the concept of governance at a distance as a new form 
of governance in relation to the book’s analytical framework. The expanding 
practices of QAE produce quantitative data about education which is said to 
allow the nation-state to extend its capacity to govern across territory and into 
the classroom. Governance at a distance relies on the production and circula- 
tion of data. This chapter focuses on the tensions and paradoxes which appear 
to make the process of governance at a distance through data less smooth than 
is generally depicted in both academic literature and political rhetoric. We first 
analyse official policy statements concerning where data are presumed to circu- 
late and for what purpose. We then use interview data to explore the paradoxes 
emerging from the political objectives of data circulation and use and from the 
paradoxical nature of quantitative data about education itself. 

Chapter 7 investigates the ways in which local authorities utilise QAE poli- 
cies to govern schools and the effects this has on their room for action. The 
analysis draws on a combination of governance theories and on the political 
frame of organisational analysis. The fieldwork data were collected in selected 
localities in Brazil, China, and Russia by means of document analysis, interviews, 
and observations. We demonstrate that the QAE instruments are reinterpreted 
locally in accordance with the pre-existing practices of quality control and 
school governance and are biased towards local actors’ political interests. High- 
performing schools can thus utilise QAE policies to draw power from sources 
such as expertise, access to agenda setting, or building networks and coalitions, 
while low performers are increasingly disadvantaged. Schools’ reputations act 
as a key to the virtuous or vicious cycles in which schools find themselves 
ensnared in the implementation of performance evaluation. Our findings also 
indicate that the room for action for those schools opposing new QAE policies 
is somewhat restricted. However, schools can practise hidden resistance and to a 
certain extent avoid the penetration of QAE tools into their internal processes. 
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Chapter 8 analyses alternative futures of QAE as expressed by Brazilian, 
Chinese, and Russian actors. We use official policy documents and our inter- 
view data to investigate how national officials and experts see the future of 
QAE in their respective contexts and the trends they perceive as preferable or 
undesirable. Our findings are set against the backdrop of the case countries’ 
socio-cultural history. We also compare the views expressed in interviews to a 
typology of future scenarios for societies and education. In addition to the envi- 
sioned “new” futures, we also find views of alternative futures which are rooted 
in domestic pedagogical traditions, including QAE practices, and express the 
desire to retain what the interviewees see as valuable in these traditions. 

Chapter 9 draws together the main findings of our research and reflects on 
their theoretical significance. We conclude that there are three dynamics in the 
politics of quality in Brazil, China, and Russia. Self-reinforcing and shared goal- 
setting reflect how QAE has overtaken quality as a goal of education policy. 
Authorising but diverted governance describes how QAE enables a parallel trend 
of authorising more governance methods, while creating increasingly complex 
systems. Destabilising and reorganising role-setting indicates how the mechanisms 
of QAE create new actors in the field, which at the same time brings instability 
to the political system because of the potential of QAE data to provoke change. 


About our research and writing 


Many of the questions covered by this book have been dealt with extensively 
in the project’s previously published or submitted articles. Reference to these is 
made in the chapters where relevant, and the respective bibliographical infor- 
mation is found in the chapters’ reference lists. 

The project consortium, led by Associate Professor Jaakko Kauko, consisted 
of three sub-projects, each responsible for investigating one of the case coun- 
tries. The leaders of the sub-projects, who are also editors of this book, were 
Jaakko Kauko for the Brazil sub-project, Professor Risto Rinne for China, and 
Professor Tuomas Takala for Russia. The individual chapters’ author teams 
worked in the sub-projects in varying roles (document analysis, interviews at 
different levels, fieldwork at the local level) and with varying total workloads. 
The researchers in the three sub-projects were 


e Brazil: Dr Vera Gorodski Centeno, Dr Helena Hinke Dobrochinski Can- 
dido, Íris Santos 

e China: Dr Johanna Kallo, Olli Suominen, Xingguo Zhou 

e Russia: Dr Nelli Piattoeva, Galina Gurova, Anna Medvedeva 


The project also benefited from valuable advice and support from three col- 
laborating professors: Viktor Bolotov (Higher School of Economics, Moscow), 
Liu Min (Beijing Normal University), and Eneida Shiroma (Federal University 
of Santa Catarina). We also invited Professor Romuald Normand (University 
of Strasbourg) to contribute to one of the chapters. 
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The project's work mode was a combination of the individual responsibilities 
first defined in the project plan and subsequently adjusted in the course of its 
implementation and frequent internal communication and intensive periodic 
workshops. Thus, the designation of individual members of the project teams as 
authors of chapters can only express the division of responsibilities in the writ- 
ing process: it does not fully reflect all the individual contributions to the intel- 
lectual process of our data analysis which underpinned the writing of this book. 
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and Xingguo Zhou 


Introduction: reflective research 


One of our research project’s assumptions is that the topic of quality assurance 
and evaluation (QAFE) is political: it is an important framing factor for educa- 
tion, a major interest for many different stakeholders, and a governance tool 
(Nóvoa & Yariv-Mashal 2003). In this respect, it is possible that our research 
will be used for political purposes, an aspect most of our research participants 
and fellow researchers certainly recognise. Self-reflection is therefore essential. 

In Chapter 1, we discussed our ontological and epistemological premises 
and how the analytical framework on which we draw, Comparative Analyt- 
ics of Dynamics in Education Politics (CADEP), directs our focus to three 
dimensions we see as relevant for an understanding of the questions raised 
in complexity studies and the approaches of political science to contingency. 
While the previous chapter addressed the “why” question of our research, here 
we open more broadly the questions “how” and “what”. There is no simple 
answer to these questions, because the shared view of scholars is that research 
is never as straightforward as research reports describe — and in this respect, 
this book is no exception. Our research journey has taken a route with paths, 
streets, cul-de-sacs, and wanderings through uncharted territories. As a research 
consortium, we have held CADEP as a compass, while continuously debating 
its interpretation. A description of this journey is needed for validity: indeed, 
sharing our journey is as important as arriving at our destination. In this chap- 
ter, we therefore chart it as fully and as critically as possible. 

We believe the key to maintaining validity in a qualitative research pro- 
ject such as ours is to adopt a reflective approach throughout. We share the 
view iterated by many scholars, but which Alvesson and Sköldberg (2009) aptly 
describe and summarise, that interpretation, and the interpretation of interpre- 
tation, is the key feature of research. They state that reflective research considers 
four elements, which we highlight here and discuss further in the following 
sub-chapters. 


* Researchers should be conscious of the interpretation made. We 
have channelled the interpretations from the outset with the help of the 
CADEP analytical framework. Despite this shared analytical starting point, 
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the following chapters use additional theoretical notions to facilitate inter- 
pretation. The framework is discussed in detail in the next section. 

* Researchers should be prudent with the systematics of the research 
procedure. Arguably, the systematics of the research procedure is the 
aspect of a research report which usually receives the most attention. We 
have devoted considerable time to developing systematic approaches and 
common goals in gathering data and indexing the research material. This 
work is described in detail in the section discussing the technical aspects of 
our research design, as well as in the appendices. 

e Researchers should be aware of the political and ideological role 
of social science research. 

e Researchers should be reflective concerning representation and 
authority in their work. These two elements of reflective research, the 
political and the ideological, and the way the text seeks to claim authority 
(Alvesson & Sköldberg 2009: p. 273) are intertwined in this book. This 
intertwining resembles the reflection offered by post-colonial theories and 
relates to us as researchers as cultural in- and outsiders and to the ques- 
tions of how we can build an understanding of Brazil, China, and Russia 
with “Western” concepts from the Global North (see Centeno, Kauko & 
Candido 2017). We construct this reflection by analysing a set of research 
narratives from the members of our group in the penultimate section. The 
political implications are already clear in our research aims as we analyse 
the use of political space; this is a question that we address in the final 
chapter. 


Our research design is best described as abductive, where theoretical and 
empirical work are complementary, concentrating on “pattern finding [which] 
is at the heart of science” (Alvesson & Skóldberg 2009: p. 7). In gathering our 
research material, the CADEP framework focused our attention on actor rela- 
tions and room for action, but our analysis was conducted more inductively, 
resulting in an investigation of the problematics which cut across the three cases 
and their inherent contextual logic. This links well to the reflective process 
of research, and the starting points of abduction reflect the fact that the same 
data open a maze of forking interpretative paths (Hanson 1972; see Alvesson & 
Skoldberg 2009: pp. 7-8). To put this differently: our research design increases 
the need for the scrutiny of validity. The four elements of reflection Alvesson 
and Skéldberg describe, listed here, provide us with tools to understand how 
deductive elements with their cultural presuppositions and theoretical under- 
pinnings or inductive analytical interpretation with its technical solutions have 
all affected our work. These aspects are elaborated in the following. 


Interpretation: comparing three dimensions of dynamics 


We continue this reflection on the nature of our research by addressing the 
question of our analytical framework in relation to those of others. Our 
research concentrates on understanding the political dynamics in QAE. To 
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posit the relevance of focusing on dynamics instead of processes of dissemina- 
tion, adaptation, and implementation, in this section, we relate our view more 
closely to comparative education theory and methodology. We call our method 
“comparing dynamics” to indicate our understanding of our research’s analyti- 
cal unit, the dynamics of education politics, implying our interest in the move- 
ment involved in education politics. 

The adoption of the dynamics of the QAE agenda as our unit of analysis 
indicates our concern to study global and local practices and policies in rela- 
tion to each other and our presupposition that they are mutually contingent 
(Schwinn 2012) and often interrelated (Schriewer 2009). While our study is 
undertaken within the borders of Brazil, China, and Russia, our approach is 
situated within the growing research corpus which problematises the tradi- 
tional understandings of country-bound territorialism and the dichotomy of 
an abstract global and concrete local and which is moving towards an under- 
standing of the production and use of space (Anderson-Levitt 2012; Dale & 
Robertson 2002; Robertson, Bonal, & Dale 2002; Werner & Zimmermann 
2006; Carney 2009; Vavrus & Bartlett 2006). 

By focusing on dynamics, we aim to transcend both conceptual global-local 
dichotomy and methodological nationalism while observing the golden rule 
of explanatory comparative studies, which is that context matters. Our com- 
paring dynamics perspective engages with what Steiner-Khamsi (2014) calls 
a “contextual comparison”, which we see as an invitation to understand the 
multi-layered comparative context. Multi-layeredness relates to how our analy- 
sis considers actors from, and the developments occurring in, different contexts 
of action, ranging from schools to international organisations. 

Studies engaged in cross-national comparisons have either shown that politi- 
cal or geographical boundaries are poor variables for explaining educational 
phenomena or asserted that local particularities are behind variations. The chal- 
lenge today appears to be to operationalise a research design that captures the 
complexity of cultural socio-political contexts which are sufficiently inclusive 
to structure, and be structured by, other contexts but remain sufficiently exclu- 
sive to (re-)engender contextual differences. As Chapter 1 indicates, we use the 
word transnational to capture the multi-layered nature of context, the practices 
and policies which develop amidst this setting, and the dynamics and spaces of 
action they produce. 

We develop a contextual comparison which considers vertical and horizon- 
tal analyses. We combine a horizontal comparison of the problematics which 
cut across the three cases with a vertical analysis of the different contexts within 
each case. It is important to note that our understanding of “contextual com- 
parison” (Steiner-Khamsi 2014) includes agency. This means conceptualising 
agency and relationality as simultaneously embedded in contextual cultural and 
socio-political situations and in self-directed actions which, although subject to 
changing power structures, enable actors’ construction of possibilities and room 
for action as indicated in Chapter 1. In understanding dynamics, we aim to 
arrive at a description of the relations of actors in these cases which will allow 


Layers of reflectivity in comparative research 21 


us to understand the patterns, limits, and possibilities of action across Brazil, 
China, and Russia. 

In Chapter 1, we described three analytical dimensions for comparing 
dynamics: the political situation, political possibilities, and the use of politi- 
cal space. In Table 2.1, we summarise how the CADEP framework steers our 
work and how it links to the questions of QAE dimension by dimension. Here, 
we must note that the different dimensions of analysis overlap in the different 
chapters, meaning that all chapters chart all dimensions to some extent. 


The political situation: a constellation of actors 
in a socio-historical context 


As Table 2.1 indicates, in analysing the political situation, we focus on how 
socio-historical dependencies create the basis for actor relations and how these 
relations are reshaped by QAE policies. The results of the Fabricating Quality 
in Education study raised questions about changes in actors in a political situa- 
tion, which revealed the interconnectedness of the buttressing and creation of 
new organisations working on statistics in Europe (Lawn & Segerholm 2011). 
The national reactions to external signals may vary (Grek & Rinne 2011) and 
depend on path dependencies (Kauko & Varjo 2008). However, it is clear the 
ascendancy of new actors is essential in reforming actor relations (e.g., Barber 
2014). 

The question of the political situation culminates with the idea of an oppor- 
tune moment (kairos) when policies can be changed (Palonen 2006). The 
dimension of the political situation leads us to analyse what the political struc- 
ture, with its constellation of actors, allows in its socio-historical and transna- 
tional context (see Simola et al. 2017). While the constellation of actors and its 


Table 2.1 CADEP framework for analysing QAE in this book 


Dimension Questions Relation to QAE 
(1) The political What is opportune in a Analysing the actor constellations 
situation specific socio-historical and their formation in the 
and transnational socio-historical situation and 
situation? the quality-related transnational 
education networks. 
(2) The political What are the political Analysing the central debates 
possibilities possibilities opened by and problematisations: how are 
prevailing discourses? quality discourses formed, and 


how are they related to what is 
considered possible in education? 


(3) The use of How do the relevant Analysing action in the space where 
political space actors exploit the QAE policies are used to reshape 
existing situations and the practices, discourses, and 


possibilities? positions of different actors. 
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relationship to the socio-historical situation has been studied with the help of 
policy process theories, the transnational dimension is an addition the develop- 
ments in comparative theories can offer. 

Indeed, the political situation dimension has been studied in political science 
literature in the context of national policy processes. Kingdon (2003) empha- 
sises the role of policy entrepreneurs in waiting for the right moment to intro- 
duce a solution to emerging problems. Baumgartner and Jones (2009) find that 
policy changes in bursts after an agenda has been populated with similar ideas. 
Sabatier (1993) emphasises the role of external factors in sub-system changes. 
All use a different interpretative framework for changes in the political situation 
but agree that socio-historical factors are important to understand the external 
limitations for politics as it evolves. They also regard the political constellations 
of actors as key to an understanding of the internal logics of a political system. 
We therefore see the relationship of these internal and external factors as one of 
the main elements in understanding a political situation (Kauko 2013; Simola 
et al. 2017). 

Theories concerning the policy process have focused on national decision- 
making, but they cannot escape comparative research’s criticism of methodo- 
logical nationalism. Whether defined by territoriality or political scope, seeing 
scales or levels of action like the national as analytical units is questionable and 
needs reconsideration (see e.g., Collinge 2005 and Issue 3 of Comparative Educa- 
tion 49,2013). We do not deny their assistance as widely used heuristic devices 
in elaborating and understanding arguments, and we therefore do not entirely 
avoid them. However, we subscribe to the idea of the mutual embeddedness of 
the local and the global, or the sub-national, national, and international, creat- 
ing transnational room for action. 

A political situation is interwoven at both levels and scales, as well as history 
(see Novoa and Yariv-Mashal 2003). Studies in global history (e.g., Ressler 
2010) and the sociology of globalisation (e.g., Sassen 2007) show the entangle- 
ment of levels and scales in the (re)construction of historical and social pro- 
cesses. They are not only connected through the mutual influence of persons 
or ideas; they also permeate each other: practices and policies pertaining to the 
global occur at the local level (Sassen 2006), and micro-activities cutting across 
localities have a global reach (Santos 2006). Space and scope, reach and impact 
are no longer defined by the scale of each occurring action. This perspective is 
useful in studying the effects of the global agenda on QAE (Chapter 1). 


The political possibilities: understanding politicisation 


The second dimension of our analysis, political possibilities, inks to how the 
discourses shape what the actors see as possible. It is generally concerned with 
identifying the possibilities in the existing discursive formations and what is 
politicised. This dimension might be described as an analysis of the discur- 
sive conditions and resources (Simola et al. 2017). When actors attempt to 
create more room for their action, they may try to politicise various issues 


Layers of reflectivity in comparative research 23 


(Palonen 1993, 2003). For example, this entails the aim of placing new issues 
on the political agenda, disputing or compromising on an existing issue, or, in 
the broadest sense, the expansion of the political arena by the introduction of 
new players or of old players as newly relevant for the arena. If an issue is not 
politicised, the action which is seen as possible is limited to the old arena and 
its patterns. 

Another, and from the comparative perspective rather fruitful, view of the 
question of political possibilities is represented by the question of problematisa- 
tion. In researching Portuguese-speaking countries, Novoa et al. (2003) intro- 
duced the idea of “interpretative community”. An imagined community was 
created through a shared understanding of the world. Later, Nóvoa and Yariv- 
Mashal claimed that the understanding of problematisations and their histori- 
cal formation was more important than essentialist comparisons of “facts” or 
“realities” (Nóvoa and Yariv-Mashal 2003). We find this idea useful for our 
analytical framework. Problematisation is a discursive structure which delimits 
what is considered relevant, and defines what is “real” and what is expected of 
different actors (see Bacchi 2012). In other words, this analytical lens focuses 
on presuppositions about causes linked to proposed solutions and on the wider 
political concerns embedded in and reflecting broad policy debates and politi- 
cal constraints (Bacchi 2012; see Kingdon 2003). 

As we argued in Chapter 1, quality has emerged as a central element of 
the shared language about education. This has happened both to problematise 
education and to resolve diverse problems (e.g., Ozga et al. 2011; Kauko et al. 
2016). This shared language about education contains a set of uniform solu- 
tions to diverse problems and contexts, and these uniform solutions rely on and 
necessitate a shared approach to problematising education. This shared prob- 
lematisation plays a significant role in making proposed solutions seem natural 
and inevitable. However, in questions of QAE, the possibilities for politicisation 
are also always present. 


The use of political space 


The third dimension of our analysis, the use of political space, is the most dif- 
ficult to analyse. This dimension seeks to describe the extent to which actors 
can capitalise on an existing political situation and political possibilities. In 
other words, the political space (with its apt German description Spielraum, 
“play/game room/space”) is shaped by the two other dimensions, and the third 
dimension seeks to understand how it is used. The use of political space may 
also be described by other metaphors, such as “the art of playing with contin- 
gency” (Simola et al. 2017: p. 18). 

Political science literature offers differing views on the use of political 
space. Prominent here is Kingdon's idea of multiple streams. Drawing on the 
garbage can model (Cohen, March, & Olsen 1972), he describes how political 
actors aim to combine policies and problems at the right moment and intro- 
duces the idea that actors (for Kingdon, policy entrepreneurs) prefer certain 
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solutions and try to find the right problems to couple these with, rather than 
vice versa (Kingdon 2003). The model has been criticised for its heuristic 
view and for its lack of understanding of historical trajectories (Zahariadis 
1999; Baugartner, Green-Pedersen, & Jones 2006: p. 963). However, it helps 
us to understand the resourcefulness and complexity of political action — how 
the first two dimensions of political dynamics (the political situation and the 
political possibilities) create the frame for political action as they determine 
actors’ room for action. 

Here a relevant contribution arises from the understanding of the social con- 
struction of space, which has been an element of comparative education since at 
least the 2000s. Larsen and Beech (2014) describe a “spatial turn” which focuses 
our attention on the relational notion and productive functions of space. This 
view holds that a “relational notion of space implies understanding that space 
not only exists in substantial, concrete, and separate forms, but as sets of relations 
between individuals and groups” (Larsen and Beech 2014: p. 199), and that the 
“global is not just some space, out there, without material basis. It is produced 
in local settings” (Larsen and Beech 2014: p. 200). We agree with this, and we 
believe the use of political space seeks to understand this productive function 
through an understanding of action or of how the space for action is produced 
through the interpretations and alterations made in the political situation and 
political possibilities. Another important contribution to the development of 
our understanding of the uses of space in comparative settings is what Carney 
(2009) describes as the “policyscape”, by which he means the similar vistas for 
action created by neoliberal policies in very different contexts. 

The focus on political action which we aim to understand in the third 
dimension of the use of political space has been especially pointed to in com- 
parative education research as a criticism of faceless explanations of processes 
such as globalisation. Dale and Robertson (2002: p. 12) remind us that the 
globalisation process always has an actor: “local structures and institutions, pro- 
cesses and practices, are crucial to, even the medium necessary for, the spread 
of global practices”. The same is true of culture according to Anderson-Levitt, 
because it “is locally produced by particular people who interact in particular 
places” (Anderson-Levitt 2012: p. 446). Ozga and Jones (2006) point out that 
travelling policies adapt to embedded ones. Space thus exists as “sets of relations 
between individuals and groups” (Larsen & Beech 2014: p. 200). In our analysis, 
the key question regarding the third dimension of dynamics concerns how the 
room for action is created as a process which is tied to the relations between 
actors as well as institutional structures. 

In our understanding, research into QAE in education has failed to take 
room for action as an empirical starting point. An exception is found in Dahler- 
Larsen’s theoretical ideas. He sees constitutive effects as the best description of 
the repercussions QAE policies have: they affect actor relations. For example, 
this is not captured by the term “unintended effects”, which connotes the pos- 
sibility of controlling the QAE process and dividing it into intended and unin- 
tended consequences (Dahler-Larsen 2011, 2012). What we seek to understand 
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is how in each of our three cases QAE policies are formed through a unique 
combination of transnational discourses, techniques, practices, and specific 
political ambitions to distinguish the particular constitutive effects of QAE 
implementation and, in this sense, how they present comparable and related 
patterns of dynamics which shed light on the room for political action between 
different scales. 


Systematics of the research procedure 


To analyse the three dimensions just described, we collected a range of research 
material from different actors in the case countries. The project’s research 
material consisted of documentary material, interviews, and observations. The 
documentary material used in each chapter is mentioned in the reference list. 
The interviews are anonymised, and they are referred to with a combination 
of country codes (BR, CN, RU), actor level and role, and interviewee number 
(see Appendix 1). Observations are referred to generally and based on obser- 
vation diaries. Data collection is tightly linked to the local conditions in each 
context, which we describe briefly. 

We selected sub-national and local cases in Brazil and Russia on the basis of 
their activity in the introduction of QAE policies and participation in interna- 
tional QAE initiatives. In Brazil, we conducted research in Santa Catarina state 
and its capital, Florianopolis, and in Russia, we studied QAE in the Republic 
of Chuvashia and its capital, Cheboksary. In China, our access to sub-national 
and local institutions was restricted, so data collection on these levels followed 
a different pattern, and the names of the case localities and organisations can- 
not be disclosed. Details about local cases in the three countries are provided 
in Chapter 7. 

Table 2.2 displays the core structure, themes, and goals of the interview guide- 
line, which was used throughout the project on all levels and in all countries. As 
well as directing the interviews, it served as a means of focusing other research 
material collection and therefore also demonstrates the means by which the 
CADEP frame was operationalised. The political situation was investigated by 
identifying actors’ roles to understand the formation of polity. The aim was to 
reach an understanding of how the different actor constellations varied depend- 
ing on the subject and related to the question of the political situation. The 
political possibilities were largely investigated based on the various themes aris- 
ing from the interviews’ introductory section. This helped us to understand 
the key questions’ problematisations. A range of questions from different parts 
of the interview guideline, especially those related to change dynamics and the 
future, was useful in analysing the political room for action. 

The complete interview guideline (see Appendix 2) contained many spe- 
cific and common questions to enable its adaptation to the different situa- 
tions in each country. The content was also tailored to the respondent’s level 
of action, context, and areas of expertise: we adjusted the interview script to 
the respondent’s profile and history. For example, where the Brazilian national 
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Table 2.2 Main themes and goals in the collection and analysis of data 


Theme Common goals 
Introduction To understand the respondent's view on quality and evaluation in school 
education 


To understand the interviewee's concrete involvement in — and its 
perceived impact on — QAE policy and practice 
Actors To understand who the main actors (collective or individual) in the 
field are and their role/action and perceived impact on quality and 
evaluation policies and practices 
To understand the position in the field of the interviewed actor and the 
connections/relations between different actors 
Change To understand changes in actor relations 
dynamics To understand the role quality and evaluation practices and policies have 
played in changes in actor relations and to identify whether and how 
they define actor relations (or vice versa) 
To understand transnational connections and their impact on national 
and sub-national policies and practices 
Future To understand actors’ perceptions of expectations and possibilities 


agency responsible for making large-scale assessments, the National Institute 
for Educational Studies and Research (INEP), was concerned, we adhered to 
the main frame when interviewing the president and departmental directors, 
but interviews with technicians were tailored according to their field of work. 
Likewise, in interviewing those working with the OECD and Programme for 
International Student Assessment (PISA), we focused on issues such as the rela- 
tions between Brazil and the OECD or the production of PISA data. Similarly, 
in Russia, there was a telecommunications agency working in camera sur- 
veillance and a media outlet creating school rankings, and in these cases, the 
interview framework was adjusted according to the area of respondents’ work. 
In the Chinese case, there was a similar logic. For example, the questions to 
policymakers focused more on the policymaking process, but when we inter- 
viewed the technicians from the national assessment centre, questions were 
more focused on their practical work in developing measurements. 
Methodological literature on interviewing those with power emphasises 
good preparation and strategies to gain access. However, as Walford (2012) 
states, the real difference here may lie in access. For example, interviews should 
be well prepared by researching both the context and what the interviewee has 
already publicly established. Although this is often seen as a feature of inter- 
viewing the powerful, they are as important for interviewing others, such as 
experts and leading managers. Contextual knowledge is also crucial for school 
interviews, for example. This was embedded in our research design: interviews 
were prepared with the help of review literature and document analysis. Fur- 
thermore, as a foreign project in three countries, our access question was not 
limited to interviewing those with power, as the next sub-chapter elaborates. 
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The Brazilian education arena is wide in both horizontal and vertical relation- 
ships. It encompasses the public and private sectors, along with third sector 
organisations, actors from different levels, and representatives of a variety of insti- 
tutions, such as government, unions, universities, professional and institutional 
associations, foundations, civic movements, committees, councils, and schools. 
As was noted in our literature review and historical analysis, there is also a tradi- 
tion of government interaction with international actors in the history of Bra- 
zilian education (see Kauko et al. 2016). We spent three periods in Brazil: March 
to July 2015, October 2015 to January 2016, and September to October 2016. 
The research data include documents, interviews, and observations. 

The Brazilian documents we analysed ranged from reports, decrees, laws, 
regulations, plans, and official discussion papers about education and QAE 
in Brazil to websites, online newsletters, Internet news, and videos by inter- 
national organisations (e.g., the Organization of American States (OAS), the 
Inter-American Development Bank (IDB), the OECD, the United Nations 
Educational, Scientific and Cultural Organisation (UNESCO), the United 
Nations Children’s Fund (UNICEF), and the World Bank), other international 
actors (e.g., the Pearson Foundation, the Inter-American Dialogue Network, 
and BRICS), the federal government, national agencies, the Santa Catarina 
government, the Florianopolis government, third sector institutions, civil soci- 
ety movements, and private organisations. For each of the three selected public 
schools (see Chapter 7) we also analysed school documents (e.g., the Peda- 
gogical and Political Plan) and data concerning these schools made available by 
INEP and private organisations (e.g., QEdu), along with websites, news, videos, 
photos, social media, and other materials provided by schools. 

Similarly, the number of interviews reflected the breadth of the Brazilian 
education field and its ease of access (see Appendix 1). All interviews followed a 
semi-structured approach based on the interview guidelines. Interviewees for- 
mally consented to the interviews, which were recorded and later transcribed. 
School interviewees were selected within each school community for their dif- 
ferent roles in respect to QAE policies and practices, and we attempted to cover 
all involved actors (administrative staff, teachers, students, and parents). We 
undertook eight weeks of observations in schools, carried out in three periods: 
1) before interviews in schools, to familiarise ourselves with their environment, 
actors, interactions, and daily routine; 2) when interviews were conducted, to 
pay special attention to the national test examination, classes, and school staff 
meetings; and 3) having completed interviews, to follow the routine and iden- 
tify the effects (if any) of the national test examination on the school environ- 
ment. We observed classes, teachers’ meetings, school council meetings, teacher 
training, external examinations, elections of school principals, end-of-school- 
year meetings, and daily activities. Additional visits were made to schools before 
the beginning of the fieldwork period and after its completion. 
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In Brazil, we were also able to make observations at meetings and events 
organised by sub-national and local governmental offices and third sector 
and civil society institutions. We observed two weeks of the State Council of 
Education of Santa Catarina’s work, which included special committee meet- 
ings, the meetings of the presidents of special committees, plenary sessions, 
and the daily routine; one working day of the Municipal Council of Educa- 
tion of Florianópolis, including the monthly meeting of municipal councillors; 
one working day of the City Council of Florianopolis, including the Educa- 
tion Committee meeting; two working days of the Movement Santa Catarina 
for Education, which included the Fourth International Seminar of Education 
organised by the Confederation of Industries in Santa Catarina (FIESC) and 
FIESC’s plenary session; and one monthly meeting of a neighbourhood civil 
society association. Informal interviews were conducted with relevant actors 
(some of whom were also formally interviewed). These informal interviews 
were neither audio-recorded nor coded but were considered as observation 
notes. 

In China, as Chapter 3 explains in more detail, despite increasing decen- 
tralisation, the educational system remains highly centralised. The Ministry of 
Education takes the leading role in organising and governing education devel- 
opment. The QAE functions of the Ministry of Education are performed by 
two sub-systems, the supervision system and the national assessment system. 
NGOs play a limited role and are required to coordinate their activities with 
the state’s requirements. At the same time, policymaking in China actively 
involves experts, especially from academia. Because of these specific features of 
the QAE system, the fieldwork in China included interviews with both gov- 
ernment actors and experts at the national, sub-national, and local levels. 

The initial research design included a larger number of interviews in China. 
However, despite official claims that the country is opening to the world, the 
attitude to foreign queries remains cautious and reserved. We were not per- 
mitted to speak to policymakers in the higher echelons of the Ministry of 
Education, so we had to compensate for this by focusing on more accessi- 
ble actors and information from open sources, such as official documents and 
media publications. 

We conducted the first period of data collection in China in June and 
July 2015, when we undertook interviews, workshops with local inspectors 
and politicians, and four school visits in a selected city. We also interviewed 
sub-national actors from NAEQ (National Assessment of Education Quality), 
the local bureau of education, and the supervision office. In the second period 
of data collection, in March 2016, we conducted interviews at the Ministry of 
Education, NGOs, IGOs, and with more researchers working as consultants 
and experts in the supervision system. We also conducted two further inter- 
views with principals. 

We collected Chinese national documents related to QAE and the main 
education policies issued by the Ministry of Education, the central govern- 
ment, and the national supervision office. The Chinese government’s five-year 
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strategic plans guide the direction of educational development. We analysed 
national plans from 1980 to 2010 and the ten-year plan for the period from 
2010 to 2020, which aims to transform schools’ examination and evaluation 
culture. We also analysed the Law of Compulsory Education (1986, 2006, and 
2015), the annual work plans of the Ministry of Education from 2007 to 2016, 
Supervision Decrees (1991 and 2012), and National Supervision Reports from 
2005 to 2015. Apart from these documents, we analysed the websites of the 
Ministry of Education, NAEQ and sub-national level bureaus of education, 
and the public speeches of national leaders and the minister. We collected bul- 
letins (in total 54 issues to the end of 2015) from NAEQ’s website, in which 
the latest assessment results are published with information about assessment 
events, training, and collaboration with foreign organisations and researchers. 
We also analysed reports from international organisations (the OECD, UNE- 
SCO, UNICEF and the World Bank, a total of eight reports dealing with 
China and QAE), and Chinese NGOs working on QAE issues, as well as these 
organisations’ websites. At the local level, we analysed the websites of educa- 
tion bureaus, archived documents, and local implementation plans for national 
policies. School websites were the main source for school document collection. 

Given the restricted access, QAE-related seminars and school visits became 
an important additional means of collecting interview and observation data 
in China. The team organised a conference to publicly discuss Chinese QAE 
with key actors, such as inspectors and decision-makers from local supervi- 
sion offices. Attendees represented a wide range of actors from the national 
to local level and included core planners and local policymakers. Information 
from lectures and discussions was recorded and collected as part of the data. 
School visits facilitated data collection at the school level. Visits typically started 
with a workshop on school QAE practices for principals and teachers. The 
schools’ presentations were followed by a discussion about the project’s inter- 
view guide, with the emphasis on “actors”, “changes”, “challenges”, and the 
“future”. Some visits were joined by local researchers cooperating with the 
schools. Visits also included school tours accompanied by teachers or princi- 
pals. Some schools prepared programmes performed by students and visits to 
classrooms and laboratories. 

In the Chinese research environment, special attention was paid to the grad- 
ual building of trust in local communities, as there were doubts about the inten- 
tions of our research and our interest in their localities, and expectations of a 
formal government letter of introduction justifying our undertaking of research 
in China. However, we met challenges in obtaining this from the Chinese cen- 
tral government (see the next section on our research journey). To secure com- 
munities’ trust, we clarified that data collection was intended only for academic 
research, emphasised our strict commitment to research ethics, and promised to 
maintain the anonymity of interviews. 

The national QAE arena in Russia comprises a range of government 
agencies and research institutions subject to or contracted by two key state 
actors: the Ministry of Education and Science and its subordinate organisation 
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Rosobrnadzor (the Federal Service for Supervision in Education and Science). 
The career paths of many QAE experts include academic and government 
positions at the national or sub-national level and sometimes involvement with 
international organisations. It is therefore difficult to draw clear distinctions 
between the roles of interviewed individuals and the levels at which they work. 
Other actors, including media, publishing houses, teacher associations, and pri- 
vate companies, perform more specific functions within the QAE system, such 
as developing rankings, evaluating national test materials, or organising training 
for schoolteachers. International organisations’ presence in Russia is currently 
limited (for more on actors, see Chapter 4). 

National-level interview data in Russia were collected in two periods, in 
June and October 2015. Data at the sub-national and local levels were collected 
in four periods in a total of eleven weeks. A one-week piloting visit to Che- 
boksary in November 2014 included interviews and two school visits and was 
followed by three periods of participant observation in schools and local-level 
interviews in May and June, October and November, and December 2015. 

Documents analysed at the national level included government programmes, 
reports, decrees, curricula, newspaper articles and news items, presentations and 
speeches by relevant individual actors, video seminars, and government agency 
press conferences. We also followed discussions at Russian academic seminars, 
in person and online. The development of the QAE system in Russia was also 
traced through a review of academic journals and books in Russian from 1990 
to 2014 (see Gurova, Piattoeva, & Takala 2015). These forums included aca- 
demic publications and expert discussions and essays by researchers, teachers, 
and members of parliament. At the sub-national level, we analysed documents 
issued by the Ministry of Education of the Republic of Chuvashia and its 
Center for New Educational Technologies, including laws, regulations, guide- 
lines, plans, and public reports. Locally, we analysed programmes, work plans, 
information circulars, the public reports of the Department of Education of 
Cheboksary and the city’s Center for Monitoring and Development of Educa- 
tion, and school regulation and action plans. We also analysed the websites of 
local organisations and schools. 

The significant influence of national academic experts on QAE system 
development, as well as their interest in our study, explains the large number 
of interviews with experts in Russia. However, we experienced difficulties in 
accessing Rosobrnadzor and its subordinate institutions, so their work was largely 
covered by document analysis and review of websites and media sources. Locally, 
schools proved the most accessible and cooperative actors, while most officials 
of municipal institutions were reluctant to participate in the study. Data about 
municipal organisations were therefore gathered either by document analysis or 
interviews arranged with the help of schools; school administrators also shared 
their perspectives on the work of the supervising authorities. Semi-structured 
interviews at all levels were based on the common interview script (see Appen- 
dix 3), which was modified and developed according to each interviewee’s 
expertise. At the school level, we also conducted several unstructured informal 
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interviews to complement our participant observation. Formal interviews were 
audio-recorded and transcribed, and written notes were taken during informal 
interviews and observation. 

Participant observation periods were selected with the intention of includ- 
ing all the main evaluation procedures in schools. Procedures were identified 
from policy documents about QAE and from schools’ work plans: state exami- 
nations after grades 9 and 11, subject Olympiads and contests, and internal 
school examinations and assessments (e.g., end-of-quarter or end-of-year tests). 
Two public schools in Cheboksary were observed, with observations of classes, 
meetings, internal examinations, and everyday activities. Interviews were con- 
ducted in these and three other schools. We also observed three municipal 
seminars for teachers. When a preliminary analysis of the collected data had 
been made, we organised a follow-up seminar for the two observed schools, at 
which our preliminary research results were presented and discussed. 


Qualitative content analysis of interviews and documents 


Qualitative Content Analysis (Mayring 2000; Hsieh & Shannon 2005; Schreier 
2012) is instrumental in systematising extensive data. Our coding frame allowed 
us to index and arrange research material in manageable pieces and helped to 
build the different foci of the research questions, such as identifying all actors 
and the descriptions of their relations. However, most of the codes were devel- 
oped in an iterative process and based on the research data. The project’s organ- 
isation meant that emphases differed slightly in the different countries. Analysis 
of documentary data and interviews differed, for example, when interview data 
were thoroughly coded and documentary data were not. 

In the Brazilian case, documentary analysis was made throughout the 
research process, taking into consideration the project framework, the prepara- 
tion of interviews, and the subsequent concept-driven and data-driven coding 
of interviews. The documents were thus analysed by the thematic issues rel- 
evant to the analyses over the research process. Thematic analysis was under- 
taken to systematise the data. It also served as background information which 
contributed to the tailoring of interview questions and literature reviews. The 
Chinese team also undertook a discourse analysis to analyse Chinese political 
problematisation in school education. In the Russian case, a thematic analysis of 
documents was undertaken. National and regional regulations, guidelines, and 
reports, as well as Russian academic journals and books, were analysed before 
conducting interviews to inform the latter. Other types of documents were 
analysed throughout the research process. 

Interviews were exhaustively analysed. The intention was to reduce the 
material under scrutiny systematically and flexibly. We thus used a mixture 
of deductive and inductive logics, i.e., the coding was both concept and data- 
driven. The starting point of the coding process was the main four themes 
guiding the interviews, which in turn were oriented by the research questions. 
This framed our coding view — what we were looking for — and frequently the 
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definition of the main categories. The coding proceeded with a data-driven 
approach, as most of the sub-categories, and even many categories, were then 
generated according to what the material provided. To give a brief example (see, 
however, a comprehensive explanation of the coding process in Appendix 2), 
in the Chinese coding, we include the main category of “actor”, which cor- 
responds to one of the four initial themes. The sub-categories “teachers”, 
“experts”, and “policymakers”, however, emerged from the material. Never- 
theless, since we decided to code all “changes” under the same code, changes in 
actors” positioning or roles were also coded under this category, while under the 
category “actor”, we included for the most part the description of the actors 
provided by the interviewees. In the three teams, the exhaustive coding of the 
interview material thus combined deductive and inductive reasoning. 


Research journeys — another layer of research data 


The different perspectives on reflectivity in this research project were described 
in this chapter’s introduction. One of the goals generated from these ideas was 
to be alert to the political and ideological roles of research, as well as the repre- 
sentations and authority produced in the study (Alvesson & Skóldberg 2009). 
These goals are demanding in a large research project. It is equally difficult to 
report them because reflectivity has occurred throughout the various project 
meetings and writing processes. Another layer of complexity is added by the 
fact that we research practices which themselves capitalise on scientific dis- 
courses, collect data, and claim to construct better policy based on firm scien- 
tific evidence. 

In this sub-section, we illustrate some of the reflective perspectives by turn- 
ing the focus on ourselves as researchers collecting research materials and ana- 
lysing complex phenomena in the cultural and political contexts of our three 
countries. A starting point is that researchers are not detached external observ- 
ers of their study but are incorporated in the field in complex and sometimes 
unpredictable and even incomprehensible ways (see Holstein & Gubrium 1995; 
Walford 2012). Researchers then undergo negotiations with others and them- 
selves about ethically sustainable or practically manageable compromises in 
positions, accesses, and roles. 

We observe that reflection on the QAE research process can illuminate the 
workings of QAE itself. We attempt to produce an interesting additional layer 
of research data. This supports the assumption that self-reflection on the ambi- 
guities of fieldwork is only relevant if it can move beyond mere revelation 
(Alvesson & Skoldberg 2009; Koning & Ooi 2013) and is “essential to the 
argument” rather than being “a decorative flourish” (Behar 1996: p. 14). Thus, 
in this sense, it enables “a fuller and deeper representation of the groups and 
communities we aim to understand” (Koning & Ooi 2013: p. 30). 

All the researchers engaged in the fieldwork wrote short personal histories 
of their fieldwork experience, considering fieldwork in the broadest sense of 
the term — not only as being “in the field” but also as fieldwork preparation, 
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corresponding with potential respondents and collaborators, and collecting 
background material. These very diverse stories, some more elaborate and per- 
sonal than others, captured memories which expressed the puzzling or satisfac- 
tory details of fieldwork, as well as complex feelings and thoughts that broadly 
responded to the question “What characterised my involvement in the accu- 
mulation of data about QAE politics in Brazil, China, or Russia?” In engaging 
in producing and analysing these texts, we hope to bridge the border between 
researcher and subject, attempting to document and make sense of some of our 
own experiences on the journey. Considering the outcomes of our research 
thus meant addressing “the processes that allowed such research to happen” 
(Bondy 2012: p. 587). In referring to these stories, we use the country code to 
indicate the team from which the author of the text comes. 


Coping with politics and bureaucracy 


Brazil underwent great political instability during the project. The political 
turbulence suftered by the government of Dilma Rousseff (January 2011 to 
May 2016) led to frequent changes in education ministers (six ministers and 
one interim minister). This instability culminated in her impeachment and 
Vice-President Michel Temer assuming the presidency. In one case, following a 
sudden change in the national minister of education, interviews already planned 
and arranged at lower than ministerial level proved possible, but higher-level 
civil servants (e.g., departmental secretaries), who changed one after the other 
during the ensuing months, remained inaccessible: 


Once back to work in Florianópolis, keen to regain the lost days, I directly 
began with follow-up emails and phone calls. But suddenly, something was 
weird: my contacts in the Ministry of Education were silent; I barely could 
pass through the receptionists, and when I managed to do so, it was only to 


get to someone’s secretary. 
(BR) 


Although the first moments of political uncertainty in Brazil affected the data 
collection phase, the subsequent ones affected the topicality of our research 
findings. A year after our fieldwork was conducted, the new Temer Govern- 
ment annulled some education reforms, including the decree on the National 
Evaluation System of Basic Education (grades 1 to 12). The construction of this 
system took years of societal effort; its termination took a single day. When it 
was annulled, we had just submitted a journal article for review which partly 
addressed the establishment of the new system (Centeno, Kauko, & Candido 
2017) and were left wondering how the gap between fast-paced political devel- 
opment and the usually slow-paced publication process might be bridged. 

On the one hand, we conducted our research amidst political instability, 
and this turmoil affected school work: interviews with teachers and school 
staff took place during a strike against the government’s attempts to remove 
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some of the teachers” rights. We also encountered difficulty in receiving ethi- 
cal clearance for school fieldwork. On the other, our general experience was 
that most Brazilian actors were easy to approach. Despite the instability at the 
school level, interviewees were quite open to participating in our research. 
Whereas the political changes made it difficult to access the politicians and 
higher civil servants who simultaneously occupied government positions, con- 
tact with politicians and civil servants who did not hold such posts during the 
fieldwork was less complicated. In Brazil, these actors remain connected to the 
political world by occupying either leadership positions in municipal, state, or 
federal governmental bodies or expert or administrative positions in tertiary 
sector organisations. We therefore received largely favourable responses to our 
interview requests from international organisations, representatives of national 
NGOs, politicians, and leading civil servants. 

The seal of official approval plays an important role in contemporary Chi- 
nese society, to the extent that in some instances, it is almost impossible or at 
best risky to proceed without it. However, the required stamp is often sub- 
merged in a sea of red tape. In dealing with government matters, the process is 
usually very time and energy consuming, especially as there is no official one- 
size-fits-all protocol concerning how one should proceed in obtaining research 
authorisation. With all this in mind, our team decided to attempt to receive 
approval at the national level, as we expected such a document to open doors 
all the way to the level of individual schools. 


We anticipated personal contacts, firm backing of the Finnish national aca- 
demic research funder — the Academy of Finland — and the ongoing expan- 
sion of bilateral cooperation in education between Finland and China to 
guarantee support from the high level of command. While the Chinese 
Ministry of Education found it hard to locate the department best fit to 
authorise our fieldwork, the unexpected change of personnel in the rel- 
evant department and the anti-Western(isation) campaign that swept over 
the education system quickly hampered our efforts. Our permission was 
denied, meaning that we would not access any ministerial official or civil 
servant working on educational matters. However, a successful cooperation 
agreement with a Chinese university secured access to academics and sub- 
national units relevant for our research topic, while the officials remained 
out of reach. 

(CN) 


The Russian military intervention in Ukraine and the subsequent annexation 
of Crimea, which began in 2014, resulted in Western and Russian sanctions and 
counter-sanctions. These unanticipated political events unfolded during our 
interviews in Russia. Military action, the Russian law restricting the actions of 
“foreign agents”, and the subsequent Undesirable Organisations Bill all fed into 
a general atmosphere of caution and uncertainty regarding how to respond to 
visit and interview requests from researchers supported by a Western university. 
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On our introductory visit to Cheboksary, the state-level officials were, however, 
open to meetings, often expressing excitement about possible collaboration 
with a country whose education system was ranked among the best in the 
world by the PISA study. Some of our official respondents had participated 
in earlier World Bank projects and remembered international collaboration 
as professionally rewarding and a personally exciting exchange of experience. 
However, we were informally reminded that the current political situation 
meant that not everyone would be willing to be associated with international 
research because of the possible personal consequences. In this complicated 
context, the team often discussed the fact that to gain trust, we needed to coun- 
teract two preconceptions. The first was that because we came from a country 
of high education quality, as measured by PISA, we would come to a country 
performing less well with a judgemental attitude and with the intention of 
comparing the “successes” of Finland with the “failures” of Russia. We felt our 
partners positioned us as experts, which made us uncomfortable and which we 
found unnecessary. Second, we had to prove to the research participants that 
their openness to our study would not cause them harm either because of the 
climate of suspicion towards foreign actors or the systems of control embedded 
in QAE politics and its diverse institutional and personal effects. 

In the Russian case, personal contacts at the national Ministry of Educa- 
tion and one of the leading universities in educational research were crucial 
to accessing some ministerial officials and educational experts in Moscow and 
provincial administrative and school staff. However, while we were able to 
interview ministerial civil servants, we failed to secure access to the agency 
functioning under the jurisdiction of that national ministry, which controls 
several sub-agencies responsible for the implementation of national examina- 
tions and other QAE procedures. Expecting Russia to function hierarchically, 
we secured a reference from a very high ministerial official, thinking it would 
open doors at the lower command level. While the HR department of the 
agency was quick to answer emails and phone calls, we were told that personal 
interviews with staff were not allowed, while written responses could be col- 
lected from relevant respondents identified by the agency itself. We were told 
that these responses would then be reviewed by the head of the agency and sent 
to us in summarised form. 

At this point, the Russian team faced an ethical dilemma: Should we give 
in to this proposal, which would provide us with relative access while risking 
our reputation as an independent research team and allowing the agency to 
use our research to engage in what we saw as direct control of its own per- 
sonnel? The situation echoed what Koning and Ooi (2013: p. 29) describe as 
transferring research “into another agenda”, which we were unhappy with. 
Following a lengthy team discussion, we decided not to engage in this process, 
shifting our main focus to the local level, documentary and media analysis, and 
interviews with academic experts and, among others, retired officials, politi- 
cians, and civil servants who no longer held politically sensitive posts. While 
we still fail to completely understand the reaction of an agency which presents 
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transparency and accountability as its main functioning logic, we see it as a 
manifestation of the closeness of the “centres of calculation” to public view, 
as well as a signal of the volatility of the actors who themselves sat amidst vast 
hierarchies and were constantly worried about job security (see Kipnis 2008). 
The head of the agency who denied us access had succeeded another young 
functionary who held his post for only a year and was dismissed for allegedly 
failing to combat high levels of cheating in examinations. Our project started 
when the current leader had held his post for a year and was perhaps mindful 
of his predecessor’s fate (see Piattoeva 2016; Piattoeva 2017). This and other 
situations alerted us to the importance of paying attention to the agendas and 
interests of active or passive research participants in relation to fieldworkers 
and the overall research topic. 

Although all three countries appear quite dissimilar in their openness to 
academic research by foreign scholars, we found our aim of interviewing actors 
who were part of official government structures and securing research authori- 
sation was very uncertain and depended on situational or external elements, 
no matter what official procedures were in place. Cultural proficiency, knowl- 
edge of the political context, and constant readiness to alter plans and seize 
sudden opportunities were indispensable in securing successful fieldwork. At 
the same time, we interpreted silence or failed access as a form of engage- 
ment which revealed something about the system we were studying, providing 
important data and findings concerning the problems of access which reflected 
the broader social milieu (see Bondy 2012). 

The bureaucratic pressure faced by the three teams during the initial stage 
of making contacts, securing the necessary research permits, and accessing 
relevant respondents was an important experience which enabled us to live 
and feel our way through some of the processes later described to us by the 
research participants as burdensome demands of various evaluation processes 
and data collection requests emanating from different stakeholders. Emotion- 
ally, we were able to relate better to the feelings of frustration, scepticism, or 
amusement such processes inflicted. Some of us experienced the bureaucratic 
processes as tricky but surprisingly seductive in their nature and in their effects 
on subjectivity. 


Manifestations of hierarchies and effects of QAE 


The schools to which we were invited on our introductory visit to the city of 
Cheboksary and which later became the prime observation and interview sites 
for our fieldwork were motivated to engage in our study by the opportunity 
to strengthen their local image and exploit cooperation with foreign research- 
ers to gain credibility with the municipal authorities. Local reactions to for- 
eign cooperation were thus far from univocal, despite the geopolitical upheaval. 
During our school visits, we sometimes felt that a show was being put on for us 
and that school principals were deliberately promoting a good image (as they 
saw it). This experience was shared by the teams visiting schools in both Russia 
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and China. As we later discovered, emphasis on positive image and reputation 
are essential to understand the impact of QAE and the reaction of schools and 
teachers to it, meaning that it would be too narrow to interpret these local reac- 
tions without consideration of the strategies for coping with QAE. Put simply, 
school staff initially interpreted external visits within the framework of quality 
evaluation and reacted to our interest accordingly. 

One of our team members reported after her first day in the field in Russia 
in the following words, alerting us, among other things, to an intriguing parallel 
between our research and large-scale international assessments: 


On Friday I met with the school principal and her deputy, who will be my 
contact person for this stay. The principal stressed that they need some offi- 
cial document — from the local Ministry or from our University — explicitly 
stating that their school participates in an international study. As the deputy 
principal explained later, their participation in international studies counts 
as the so-called project activity and gives them privileges in regional rank- 
ings, and even additional funding. So that is their main motivation to take 
part in our study. When schools participate in TIMSS [Trends in Interna- 
tional Mathematics and Science Study] or PISA they get the same bonus. 

(RU) 


Apart from obtaining recognition from the educational authorities, the Rus- 
sian schools in our study could exploit their involvement in our research pro- 
ject in other ways. One of us made observations at the end of the school year 
and attended several graduation and end-of-year events. These included per- 
formances prepared by students and teachers and speeches from “honoured” 
guests, e.g., local entrepreneurs, members of the municipal council, and local 
administrators. The audience included school staff, graduates, and their families 
and friends: 


At the first graduation event that I attended I was invited to sit close to the 
stage. I prepared to take notes, as I normally did at lessons and meetings, 
and did not expect any special attention. Suddenly, I heard my name and 
position (“a researcher from the University of Tampere in Finland”) from 
the stage — I was listed among the honoured guests who were later to greet 
the graduates with some speeches. It was a complete surprise, and I had 
to quickly think about what to say. Later during my stay, another event 
was organised with even greater publicity ... The purpose of the event 
[“The Stars of the School”] was to congratulate, honour and reward best- 
performing school students: those who had high academic achievements, 
won local learning competitions, led student organisations, etc. A deputy 
principal called me in advance and asked me to prepare a small speech. 
On the one hand, my position as an international researcher made me, in 
school administrators’ view, a fit person to address the students. On the 
other hand, as I felt it, the school demonstrated its prestigious “international 


38 Jaakko Kauko et al. 


cooperation” to a wider audience, including parents and local authorities 
in the audience. 
(RU) 


Teachers in Brazil were very enthusiastic and spontaneous when they were 
interviewed, which we interpreted as their sense that their voices were sel- 
dom heard. They appeared to share their thoughts without reservation and 
often seemed deeply animated by the topic of our research. Their emotional 
reactions to the questions might be read as shedding light on the very per- 
sonal, troubling impact of some of the QAE-related policies on schools’ daily 
work and the fulfilment of the duties teachers considered professionally and 
ethically desirable. Reflecting on individual interviews with teachers in general, 
some of our team members recalled situations in which teachers began to cry 
when they expressed their frustration and helplessness in dealing with mount- 
ing paperwork, which distracted them from “teachers’ work” (Russia), and the 
unproductive education reforms and teachers’ shrinking motivation (Brazil): 


She told me she had always welcomed reforms, as she thought they would 
be the outcome of well thought problem solving, and she would cooperate 
to implement them. However, at the moment, for the first time, she said 
she is hopeless. . . . “Politicians are not trying to make anything better for 
anyone [there is a lack of resources and lack of will], but now even teachers 
are so demotivated that they are not coping with the education mission.” 
She links this demotivation to the vulnerable career of educators, who are 
badly paid and need to work several shifts to provide minimum income 
for their families. She told me they are always tired, they are not respected 
in the society and, recently, neither in the classrooms. She acknowledged 
that they are held accountable for so many things concerning teaching and 
learning, but complained that, at the same time, they lack autonomy over 
their jobs and are blamed for everything that goes wrong at the school. 
(BR) 


Although the researchers were caught off-guard by interviewees’ emotional dis- 
tress, they also felt that by asking difficult questions and allowing the respond- 
ents to express their anxiety, they were able to give something in return — a 
momentary emotional relief and a sense that thoughts and feelings mattered: 
“In the very moment when their supervisors were putting pressure on them, 
demanding good examination results and completed end-of-year reports, I was 
there to listen and sympathise” (RU). 

In China, in group interview situations, teachers’ voices were often silenced 
or overshadowed by senior staff members answering questions on their behalf. 
This supported the idea that there was a silent rule that speaking priority fol- 
lowed the hierarchical order: senior staff, senior teachers, and junior teach- 
ers. If not addressed directly, most junior teachers waited for senior members 


Layers of reflectivity in comparative research 39 


to answer first. Group meetings’ interior dynamics seemed to characterise the 
hierarchy of the teaching staff. 


In one of our group meetings with both senior and junior personnel, 
new teachers always had to wait for the senior teachers to talk first unless 
directly nominated to answer the question. We addressed a question to 
a new teacher, and he/she immediately suggested that a senior teacher 
should answer it, explaining to us that the senior teacher might know bet- 
ter. This younger colleague only started talking after the senior teacher 
declined to answer and recommended that this new teacher should answer. 

(CN) 


Our observations suggested formal promotion led to differences in the staff 
hierarchy. Senior staff members had once been teachers. When they were pro- 
moted to the school administrative or management level, they started to behave 
and were treated as experienced and authoritative teachers, especially by newly 
recruited teachers. Levels of experience and status were transformed into infor- 
mal authority. However, there were some variations. For example, in our group 
interviews, we encountered cases where senior staff answered one question and 
young teachers challenged the answer with a different perspective. 

These authority dynamics changed when the group attendees included staff 
from supervisory bodies or government. In such situations, inspectors became 
the authority and retained the right to answer questions first. If there was a 
difference of opinion, school participants tended to keep silent. For example, 
there was one case where our research team asked about the challenges of 
QAE for teachers, and an inspector answered that “good teachers don’t have 
problems, only bad teachers do” (CN-S-05), silencing the whole group and 
preventing further answers. 


Conclusion: the research’s room for action 


We started this chapter by emphasising the need for reflection in research and 
placing our work more systematically within the CADEP framework. Having 
discussed the practical work of our analysis, we turned the analysis on ourselves 
and presented an account of our multifaceted research journeys. 

We have thus demonstrated that the conduct of this research could not 
escape each case’s basic political dynamics. In other words, we felt the limits of 
our own room for action in these three contexts. We aimed to learn something 
from this experience about the phenomena we were studying. 

The changes in our contexts’ political situation were described by revisit- 
ing difficulties in accessing the field and observing that these challenges were 
entangled in the changing politico-bureaucratic conditions. What we could do 
and how others perceived us were influenced by the ways in which the study 
of QAE cut across QAE procedures and their impact. This crystallised the 
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ambiguous political and ideological role of social science research in general 
and research focused on quality politics in particular. 

Our room for action in concrete work and research was linked to multi- 
faceted questions of authority. As researchers, we were often concerned about 
how our research might jeopardise participants’ positions. This represented a 
serious challenge to our work, which we addressed with a consistent anonym- 
ity procedure. We could also see power relations at play during our fieldwork, 
in which some voices were silenced and others amplified. However, we were 
also able to shed light on the ways participants could find avenues for momen- 
tary empowerment through our research. This sometimes happened when our 
research team’s presence was used as a sign of quality or interviewees used our 
interviews as moments of reflection and inspiration or simply of emotional and 
professional time off as some heavily burdened teachers did. 

As researchers striving for access and recognition, we experienced state 
authority and bureaucracy which replicated the ways in which QAE systems in 
each country worked: the chances of gaining access, the strength of the state in 
controlling the lower levels of bureaucracy, and the way top-down management 
worked. The research journeys also indicated that QAE as a governance tool 
left room for action for those it affected. As Chapter 7 demonstrates in more 
detail, schools have become experienced in playing the game of quality and can 
use an international research project to their advantage. 

In the following chapters, we start to analyse the dynamics in the politics 
of quality in Brazil, China, and Russia. To assist our reader in navigation, in 
Figure 2.1, we sketch an approximate map of how intensively the different 
chapters deal with the various dimensions of our analysis. Chapter 3 describes 
the relevant aspects of the changing political situations in each country or the 
historical-social developments which underpin their respective paths to QAE. 
Chapters 4, 5, and 8, analysing the national level, lean slightly more towards 


(1) The political 
situation 


CH6 CH7 


(2) The political (3) The use of 
possibilities political space 


Figure 2.1 Dimensions of the analytical framework (CADEP) and the book’s chapters 
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analysing the political situation, whereas Chapter 6 leans more towards the 
political possibilities created in data use, and Chapter 7 scrutinises schools” 
political space. In Chapter 9, we bring the idea of dynamics to bear on under- 
standing how these analyses of the three dimensions can be drawn together to 
construct arguments about the dynamics of QAE policies. 
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interest in quality assurance 
and evaluation 
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Galina Gurova, Johanna Kallo, and Xingguo Zhou 


Introduction 


This chapter explores and outlines the historical background of understanding 
the development of quality assurance and evaluation (QAE) policies in Brazil, 
China, and Russia and thus also prepares the ground for the problematics the 
subsequent chapters analyse. In doing this, we build largely on the previous 
research literature, albeit occasionally supplementing our analysis with primary 
documents, such as media reports, policy documents, and parallels found in our 
interview material. As the following chapters elucidate, although at the level 
of practical implementation, the countries’ experience differs markedly, at the 
level of political projects and rhetoric (see Chapter 6), they share a growing 
interest in QAE. In building the case for future chapters, our foremost task 
is therefore to ask how these three countries, which until recently developed 
separately, came to share quite similar domestic expectations and interest in 
QAE. Drawing on politico-socio-historical approaches, we outline the histori- 
cal trajectories and antecedents of QAE policies in each national context. Our 
ambition to trace the socio-historical roots of the phenomenon at hand also 
resonates with the book’s broader framework, as uncovering the political situ- 
ation is one of the three analytical dimensions of the CADEP framework the 
preceding chapter introduced. 

More concretely, we approach our topic by presenting three case-by-case 
chronological narratives. As QAE policies are linked to some of the megatrends 
of the latter half of the twentieth century, which has seen growing possibilities 
of transnational data flows and an increasing emphasis on economic efficiency 
in public services, the focus of our analysis will be on post-World War II devel- 
opments. This historical analysis will extend to the present and the challenges 
the countries are facing. We summarise this analysis with an integrative sum- 
mary in which we draw together some similarities and differences in the devel- 
opmental paths towards the countries’ shared interest in QAE. As a postscript, 
we conclude our work with an examination of signs that the three coun- 
tries are increasingly engaging with the international community and with 
each other, especially in matters related to education, suggesting possible future 
research avenues. 
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Path dependency and contingent conjectures 


Chapters 1 and 2 discussed the importance of understanding the path depend- 
encies deriving from our ontological presuppositions concerning complexity 
and, more closely, how the political situation is analysed as a constellation of 
actors in a specific socio-historical context. In this chapter, with our focus on 
processual perspectives in tracing the historical paths leading to the introduction 
of modern QAE policies, we build theoretically on the notion of path depend- 
ency, which essentially asserts that present and future choices are made within 
the constraints set by past choices (O’Sullivan et al. 2006). In the broadest sense, 
path dependency refers to the very vague notion that “history matters” (Pierson 
2000a; Mahoney 2000), but our aim is to go deeper by operationalising the 
term to show how history matters. Although the concept is used in various ways 
in different research traditions, all path-dependency perspectives stress several 
relevant arguments which help us to understand the broader context of devel- 
opments, such as the interconnectedness of events and the relevance of timing, 
showing that important developments are frequently the outcome of the earlier 
breaking points resulting from particular conjunctures (Baumgartner & Jones 
2009; Capano 2009). 

All path-dependency perspectives also share an interest in contingent events, 
which in turn place these ideas within a collective of several theories that 
together form the complexity theory. This emphasis on contingency further 
connects our starting point with the book’s broader theoretical approach. Dif- 
ferent types of path-dependency analysis focus on different types of sequences 
of events. We subscribe to the notion of reactive sequences, which is a form of 
analysis especially suited to historical narrative accounts (Mahoney 2000). These 
reactive sequences are temporally ordered and causally connected and reactive 
in the sense that events are at least partly reactions to — and thus dependent 
on — earlier events on which the historical event setting the chain in motion is 
at least partly contingent (Mahoney 2000; Pierson 2000b). To avoid the prob- 
lem of infinite regress (1.e., the endless pursuit of contingent events or reasons), 
conjunctures — intersection points between two or more prior sequences — are 
often treated as the initial contingent occurrences (Mahoney 2000). Therefore, 
we concretely set out to uncover the sequence of events which have resulted 
in the contingent conjuncture that has since led to the three countries’ current 
interest in QAE. 


Individual paths towards quality assurance and evaluation 


Brazil: rapid pace towards assessment, slow pace towards quality 


Brazilian education is closely connected with the country’s socio-economic 
and political development. During the centuries of colonisation (1500-1822) 
and decades of empire (1822-1889), schools were scarce and primary education 
was a luxury of the elite. The First Republic (1889-1930) introduced a federal 
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system and consolidated educational decentralisation. The first Brazilian Con- 
stitution (1891) recognised the states as the main providers of basic education 
and made the federal government responsible for higher education (Romanelli 
2007 [1978]). 

Historically, federalism became associated with the democratisation and 
decentralisation of political power (e.g., Souza 2002). Democratisation and 
decentralisation, in turn, became discursively connected with regionalism. 
On the one hand, because of Brazil's huge area, “regionalism” describes the 
socio-cultural and linguistic diversity of its geographical regions. Decentrali- 
sation became tied to the regional diversity which shaped Brazilian identity. 
On the other, “regionalism” refers to regional inequality and disparities in 
population concentration, resources, and socio-economic and educational 
development. Decentralisation became tied to federalism, a political system 
which allows cultural heterogeneity while seeking to counter-balance, or 
accommodate, socio-economic heterogeneity (Souza 2002). However, by 
1891, federalism was benefiting the autonomy of the states’ oligarchies, weak- 
ening the federal government, increasing regional disparities, and creating 
disparate policies (Romanelli 2007 [1978]). The states failed to expand edu- 
cation policies, and gaps in education increased. The 1920 census showed a 
72% illiteracy rate among the population over the age of five (Haddad & Di 
Pierro 2000). 

A growing urban middle class and an engaged intellectual community, with 
social discontent and the decline of the oligarchies, paved the way for the 1930 
revolution, when there was a civil-political restoration (1930-1937). A syn- 
ergy between educationalists’ vigorous demands and democratic efforts led to a 
political commitment to education planning. The 1932 reform modernised sec- 
ondary education (Dallabrida 2009). The 1934 constitution designated national 
education as the responsibility of the federal government, and the Ministry for 
Education and Health was created. Two key institutes were established: in 1934, 
the Brazilian Institute of Geography and Statistics (the IBGE, until 1937 the 
INE), and in 1937, the National Institute for Educational Studies and Research 
(the INEP, previously the INP). The IBGE provides Brazilian socio-economic 
indicators, census, and other statistics analysis; the INEP is responsible for all 
large-scale assessments and other education studies and data. 

This intellectual and socio-political unity rapidly broke down, and both the 
education and political fields were fragmented in conflicting positions. In 1937, 
President Vargas installed a dictatorship (1937-1945), and the 1937 constitu- 
tion reduced the government’s responsibility for education. However, Vargas 
continued to see education as an instrument for the construction of the nation 
(Peixoto 1995), and the corporatist regime maintained ties with certain educa- 
tionalists (Centeno 2010). The 1942 reform sought to respond to rising social 
demands for post-primary education and the requirements of industrialisation. 
The country still struggled with the provision of universal access to primary 
education. Despite growing enrolment rates, educational provision fell far short 
of meeting social demands (Oliveira & de Araújo 2005). 
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The Second Brazilian Republic (1946-1964) re-established civil rights. The 
1946 constitution reaffirmed the government's responsibility for national edu- 
cation and legislated for compulsory free primary education. Populist govern- 
ments emphasised a developmentalist industrialisation project, and education 
efforts concentrated on technical training at the expense of primary educa- 
tion (Centeno 2010). The states expanded their education systems; appropriate 
pedagogical-administrative planning was, however, absent (Nardi, Schneider, & 
Rios 2014). 

At the end of the 1950s, educationalists identified major education prob- 
lems, some of which are still present: insufficient school expansion; lack of 
equal opportunities; high dropout and retention rates; inadequate infrastructure, 
materials, and curricula; and a largely untrained teaching body (Moreira 1956). 
In 1961, after thirteen years of lively debate, the first Education Law established 
common national guidelines. The main education indicators were the national 
census and statistics concerning school enrolment, dropouts, and retentions. 
QAE practices and policies were neither problematised nor envisaged until the 
1970s, as elaborated elsewhere (Kauko et al. 2016). 

Populist governments tried to accommodate social demands and allowed a 
vibrant education community but faced strong political and socio-economic 
instability. A society fragmented into sectors with divergent interests (Ianni 
1975) was the backdrop for a military coup and dictatorship (1964-1985). The 
mulitary governments restricted civil and political rights. Education initiatives 
were tailored to meet the government’s ideological purposes and economic 
requirements. The government’s alphabetisation programmes (MOBRAL) 
were particularly well-received by international organisations like the United 
Nations Educational, Scientific and Cultural Organisation (UNESCO). 

Nevertheless, the lack of or insufficient planning for mass education raised 
serious issues at the state level. In the 1970s and 1980s, the states started to 
develop performance assessments to collect information about issues as diverse 
as teacher training, curricula, school materials, and students’ progress (Gatti 
1987). The World Bank collaborated with many of these state programmes 
(Gatti, Vianna, & Davis 1991). 

The Ministry of Education also supported programmes which improved 
school access and pupils’ performance in the country’s poorest regions (Horta 
Neto 2007). The most famous was the Northeast Basic Education Project 
(EDURURAL), which the Ministry of Education had planned in 1977 (Horta 
Neto 2007) but which was only implemented in 1980 with a loan from the 
World Bank. The first large-scale assessment involving more than one state 
was developed within this project in response to a request from the World 
Bank’s project evaluation (Gatti, Vianna & Davis 1991). In 1984, the Ministry of 
Education and the World Bank began conceiving the Second Northeast Basic 
Education Project, but the agreement was only signed after ten years of nego- 
tiations (Horta Neto 2007). Until the mid-1980s experiments with learning 
assessments in Brazil were discursively and pragmatically inconsequential and 
were still neither framed nor understood as QAE practices and policies. Indeed, 
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the notion of quality was rather contested in the educational field because of its 
historical association with elitist education and political conservatism (Centeno, 
Kauko, & Candido 2017). 

With the end of the military dictatorship in 1985, socio-political move- 
ments and education debates returned (see Chapter 4). Brazil faced economic 
adjustments (Wirth 1997) and pressure from the international system, including 
the World Bank and the International Monetary Fund, to structurally reform. 
The transitional government began a managerial reform of public administra- 
tion. Total quality management emerged as a particularly good solution for the 
country’s managerial problems (Longo 1996). Although such programmatic 
incursion into education had little practical impact, it triggered an important 
public debate. While the education community strongly contested its appli- 
cation to education, policymakers, economists, and sociologists embraced the 
educational “total quality” perspective (Centeno, Kauko, & Candido 2017). 
This discussion brought the notion of education quality to the fore. A new 
political discourse, in which quality in education was associated with assess- 
ment, emerged. 

In the 1980s, instead of substantial reforms, the states attempted to regular- 
ise school progression (in other words, without retention) through measures 
such as automatic grade progression, the provision of acceleration classes, and 
the reorganisation of studies into cycles (Franco et al. 2007). These ad hoc and 
palliative policies compromised traditional statistical indicators and weakened 
trust in official statistics (Oliveira & Araújo 2005). From the late 1980s, the 
United Nations Development Plan (UNDP) promoted the improvement of 
education information systems at both state and national levels. Projects in 1986 
focused on managerial training for basic education, support for methodologi- 
cal development, and the provision of the State Secretariat of Education with 
the necessary equipment for data collection and analysis, besides staff training 
(Coelho 2008). The UNDP has been playing an important role in collecting, 
organising, and disseminating data, including education data (this culminated in 
“The Human Development Atlas in Brazil”: see www.atlasbrasil. org. br/2013/ 
en/home/). 

In 1987, the INEP designed a programme of external assessment to evaluate 
pupils in public schools (Vianna 1990) and provide information to the states 
about learning problems. The Ministry of Education and some states commit- 
ted to this programme (Horta Neto 2007), but the project did not affect public 
policies as expected because of constant personnel changes in the Ministry of 
Education (Gatti, Vianna & Davis 1991). 

Nevertheless, this allowed the INEP to build its own expertise (see Chap- 
ter 5). This experience, together with the knowledge developed by the Ministry 
of Education in EDURURAL, facilitated the creation of the first nationwide 
large-scale assessment in 1988 (Horta Neto 2007): the Evaluation System of 
Public Primary and Lower Secondary Schools (SAEP). Resources for its plan- 
ning and implementation were still sourced from a loan agreement with the 
World Bank (Coelho 2008), but the federal government postponed its launch 
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for financial reasons (Gatti, Vianna & Davis 1991; Horta Neto 2007). The 
SAEP provided the technical grounds for the assessment of basic education 
(Bonamino & Sousa 2012). It was the first in an avalanche of national and inter- 
national assessments which progressively covered the whole education system 
(see Kauko et al. 2016). 

A strong democratic movement progressively and irreversibly took hold in 
the 1980s, leading to the 1988 constitution and the first direct presidential elec- 
tions in 1989. Federalism, decentralisation, and democracy were again entan- 
gled in constitutional debate. The 1988 constitution reinforced federalism by 
empowering municipalities with responsibility for education, but it ignored its 
financial and political impact and fuelled federal conflict (Araújo 2010). Decen- 
tralisation resumed and strengthened. Where decentralisation to sub-national 
governments is concerned, Brazil is considered the most decentralised higher- 
income country (Souza 2002). The national education scheme was reorganised 
according to the government level, and school autonomy was progressively 
enforced (Meade & Gershberg 2008). 

An ambiguity concerning decentralisation is reflected in the proliferation of 
attitudes which see it as an obstacle or even as a factor in Brazilian education 
disparities (see Romanelli 2007 [1978]; Alves 2007; Dourado 2007; Franco 
et al. 2007; Goncalves & Franca 2008; Sobreira & Campos 2008; Souza & Costa 
2009; Durham 2010). However, for many who support national standard meas- 
ures such as a core curriculum, decentralisation is still strongly tied to diversity, 
and references to a unified or centralised education system are therefore not 
only mistaken but also carry negative political connotations, which all actors, 
regardless of their position in the education arena, seem keen to avoid (as our 
interviewees vividly confirmed). 

The text of the 1988 constitution also contributed to the understanding of 
education as an individual and social right. It guaranteed the universalisation of 
access to free compulsory basic education, equality of educational opportunities, 
and non-discrimination. However, it associated education quality with educa- 
tion assessment (Freitas 2004; Coelho 2008; Gusmao 2010). The assurance of 
quality became a central aim of education policy. Some held that the assessment 
of education quality by the government was a means of allowing private educa- 
tion to continue, while attempting to control it (Coelho 2008). The assessment 
of private schools logically implied the assessment of public schools (Freitas 
2004). The SAEP was transformed into the SAEB (Evaluation System of Basic 
Education, see Chapter 7), “basic” replacing “public” to include assessment of 
private schools (Horta Neto 2007). The implementation of the SAEB in 1990 
was assisted by the UNDP and funding from the World Bank (Coelho 2008). 

In 1996, the National Education and Framework Law (LDB) entrenched 
the decentralisation of education, which was successively redefined by fur- 
ther amendments (see Law 9.394). Currently, municipal systems are broadly 
expected to provide pre-school and elementary education (for 1- to 14-year- 
olds), while the state systems are expected to cover the remaining compulsory 
education (for 15- to 17-year-olds). The federal government is responsible 
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for providing the necessary technical and financial backing to federal units. 
However, municipal and state systems frequently overlap, and this (dis)articula- 
tion between the three levels still presents many problems (Candido, Kauko, & 
Centeno submitted). Educational decentralisation has neither simplified educa- 
tion planning and political-fiscal mechanisms nor reduced inequalities between 
federal units, among the population, or even among schools (see Oliveira & 
Santana 2010). 

UNESCO and other international organisations supported the Brazilian 
educational reforms of the 1990s. The 1990s World Declaration on Education 
for All (EFA) and UNESCOS education quality agenda greatly influenced the 
education debate and the implementation of QAE mechanisms. The UNE- 
SCO Regional Bureau for Education in Latin America and the Caribbean 
(OREALC) hosted the Latin American Laboratory for the Assessment of the 
Quality of Education (LLECE), which began to conduct regional large-scale 
assessments in 1997. 

The SAEB went through several technical changes. It was reformed (1995), 
consolidated (2005), and enlarged (2013). The UNDP continued to enable 
these methodological developments and the acquisition of expertise by financ- 
ing INEP staff training (in Brazil and abroad) and facilitating international 
technical cooperation (see Coelho 2008). The UNDP employed — and still 
employs — Brazilian technicians and officials working in the INEP for example 
(interview data). Despite the critiques of data production by the national gov- 
ernment (see Chapters 5 and 7), assessments are frequently justified in the edu- 
cation arena as a counterpoint to the complex decentralised Brazilian system. 
Our interview material suggests that inequalities and disparities, with a lack of 
information and central control resulting from strong decentralisation, were the 
basis for the restructuring and expansion of QAE education policy between 
2005 and 2009 (see Kauko et al. 2016). 

Another fundamental development in QAE was the creation of a major index 
from the school census (statistical data) and SAEB data (students’ achievements), 
the Development Index of Basic Education (IDEB), which is regarded as the 
main indicator of Brazilian education quality. The IDEB was developed along- 
side the 2007 Education Development Plan, which set the goal of achieving the 
OECD’s Programme for International Student Assessment (PISA) mean score 
by 2021. The recent National Education Plan (PNE 2014-2024) reiterated this 
goal. The OECD and ETS (the Educational Testing Service leading the PISA 
consortium) trained Brazilian technicians in the early 2000s (interview data). 
Brazil has been vice-chair of the PISA Governing Board since 2013. 

In the broader political and socio-economic context, Brazil entered the 
twenty-first century with a promising framework. For the first time since the 
end of dictatorship, Cardoso’s stable democratic government (1995-2002) was 
in power. Cardoso tackled economic restructuring, implementing managerial 
and efficiency reforms in all sectors of public administration (Derqui 2001). 
Lula da Silva’s Workers’ Party government, which succeeded Cardoso, gov- 
erned in a context of progressive socio-political growth (2003-2010). The Lula 
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government ensured an unprecedented political continuity in Brazil. However, 
the socio-economic and political situation deteriorated rapidly during the next 
Workers’ Party government of Dilma Roussef (2011-2016), which culminated 
in impeachment. Socio-political contestation and economic recession have 
continued under the Temer provisional government (2016 to present). Labour 
and social security reforms are internally highly controversial, although they are 
praised by international economic agencies and financial markets. The reform 
of secondary education provokes diverging opinions. Currently, the minimum 
wage is US $297 and the unemployment rate continues to grow (hitting 13.7% 
in March 2017, IBGE). The media has described people's growing difficulties 
(e.g., Phillips 2017). 

The Brazilian socio-political pendulum swung dramatically between dem- 
ocratic and authoritarian governments until the mid-1980s, but despite this 
socio-political turbulence, there was an expansion of education opportunities 
(e.g., Oliveira & Araujo 2005, Klein 2006, Oliveira 2007, Gouveia & Souza 
2013). Since the 1990s, Brazilian education has been subject to relentless QAE 
policies. Although many — if not all — the problems experienced between the 
1950s and 1980s have persisted, educational progress has occurred. Despite 
the current turbulence, this is likely to continue. When we were in the field 
(2015/2016), Brazilian opinions about the future on the national level were 
hopeful (see Chapter 8), while locally they were very mixed (interview data). 
Currently, a broader pessimism can be sensed in informal local feedback from 
interviewees, and it is also noticeable in many media interviews with experts 
holding contrasting political and educational views (e.g., Daniel Cara, Folha 
de São Paulo, 10 October 2017; Ricardo Paes de Barros, Istoé Dinheiro, 8 
September 2007). 

The same education problems persist but with changes in their depth and 
breadth. As one interviewee concisely stated, improvements occur continuously 
but at a slow pace and appear to frustrate progress — always taking place but 
never matching initial expectations. 


China: imperial legacies and the needs of the party-state — “QAE 
with Chinese characteristics” 


The Chinese education system has been strongly influenced and shaped by 
Chinese cultural and intellectual traditions, especially Confucianism (Lee 2000). 
During the millennia-long imperial era, the state's direct involvement in educa- 
tion was very limited, and education was for the privileged few, even if formally 
everyone could achieve social mobility through the open civil service imperial 
examinations (keju), which also served as a rudimentary assessment system for 
future government officials (Han & Yang 2001; Postiglione 2011). To this day, 
this heavy emphasis on examination performance looms over the education 
system, complicating many efforts at reform. During the imperial era, there was 
also a system called the inspection of learning (shixue), under which emper- 
ors and local governors acted as education inspectors (Hong 1991). After the 
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abolition of the imperial education system in 1905, which was followed by 
the fall of the last imperial dynasty in 1912, foreign, and especially Western, 
supervision practices began to be studied and borrowed (Han & Yang 2001; 
Lee 2000). Further development, however, was stalled for decades because of 
internal conflict, war, and social turmoil (Han & Yang 2001). 

It was this context of a decentralised elitist system with a huge illiterate 
population which allowed the Chinese Communist Party (CCP), established in 
1921, to grow in influence (Postiglione 2011). After the rural-poor supported 
communist victory over the nationalists in the civil war the People's Republic 
of China (PRC) was established in 1949. The decentralised and largely private 
education system was nationalized, and the Ministry of Education took over as 
the highest central authority dealing with education issues (Hong 1991; Meis- 
ner 1999). This centralised model of education governance largely followed 
the Soviet model (Meisner 1999; Postiglione 2011). In the following decades, 
education policies fluctuated between the moderate and radical, mirroring the 
general political situation. In the most radical phases, even hinting about learn- 
ing from Western capitalist countries was an offence approaching treason. To 
cater for the needs of the rural poor who had helped the Communist Party 
gain power, in the first decades of communist rule, education served mainly 
ideological, economic, and political requirements, and egalitarianism was the 
expected targeted standard (Chen Cravens, Chu & Zhao 2011; Han & Yang 
2001). Throughout the country, education was uniformly organised and politi- 
cal orthodoxy was prioritised. The centralisation of the education system also 
covered matters related to supervision. An inspection office (shidaosi) took over 
the task of assisting education governance through the inspection of schools 
(Hong 1991). However, despite the communist victory in the civil war and 
the subsequent nationalisation of education, uncertain times continued. The 
restored supervision system and the rest of the education system were severely 
disturbed by the turmoil of the Cultural Revolution — the most radical of 
the early political campaigns of the People’s Republic of China. According to 
the extreme leftists’ view within the Chinese Communist Party, the revolution 
needed to be continued and deepened, as this task had been neglected after the 
establishment of the “New China”. 

It was only after Mao’s death in 1976 that Deng Xiaoping, the country’s 
new de facto paramount leader, began to restore the education system after the 
turmoil of the Cultural Revolution (Hong 1991; Postiglione 2011). Under 
his reforming leadership, China adopted an economic reform policy known 
as “Reform and Opening Up” (gaige kaifang). Opening the Chinese economy 
and society to foreign trade and ideas was essential to this programme so that 
“socialism with Chinese characteristics” — an economic system combining a 
traditional socialist politico-economic system with a market economy — could 
be established (Meisner 1999). In reforming the Chinese economy with the 
help of foreign expertise — which now included ideas, practices, and products 
from the West — the revitalisation and reform of the education system played 
a central part. Education became a strategic priority because of its crucial link 
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with the promotion of national development and its furthering of the state's 
economic aspirations (Chen Cravens, Chu & Zhao 2011). In a country where, 
despite the progress made during the Mao era, even basic infrastructure and 
access to education were severely lacking, efficiency and quality of education 
were naturally defined in quantitative terms. Later the focal points of this quan- 
titative improvement, with its twin goals of universalising nine-year compul- 
sory education and eradicating illiteracy, became known as the “Two Basics” 
(liangji) (Postiglione 2011). 

To ensure that these quantitative improvements would succeed, the supervi- 
sion system also had to be re-established. Consequently, in 1983, the Minis- 
try of Education released a formal document calling for the re-establishment 
of education supervision in all provinces. The legal framework guiding the 
work of supervision has since been completed (Yang 2001; Dahlman, Zeng, & 
Wang 2007). Under this model, the supervision system (dudao) has two basic 
purposes: administrative (duzheng) and education inspection (duxue). Admin- 
istrative inspection is tasked with ensuring that local governments fulfil their 
responsibility to provide the necessary funds, facilities, and resources for the 
development of education. Education inspection in turn is intended to ensure 
that schools follow relevant education laws and policies (Huang 2009; Jin 2004; 
Yang & Guo 2005). A four-layered supervision network extending from the 
national to sub-national city and county levels has slowly been implemented 
(Hong 1991; Lee, Ding, & Song 2008). 

With the help of the re-established supervision system, China was able to 
achieve most of the goals of the “Two Basics” by the end of the twentieth 
century — a feat rarely achieved by other developing lower-income countries 
(Chen Cravens, Chu & Zhao 2011; Postiglione 2011). Meanwhile, economic 
reform had succeeded in launching the People’s Republic of China on an 
unprecedented trajectory of double-digit economic growth, which helped pull 
hundreds of millions of Chinese out of poverty — a remarkable feat by any 
standard (Meisner 1999). However, exposure to Western ideas and market- 
oriented reforms also brought new problems. Income inequality, inflation, and 
corruption increased, and demands for political reform strengthened during the 
1980s. The decade’s ensuing political unrest culminated in the 1989 crackdown 
on the Tiananmen Square protest in Beijing. The Chinese Communist Party 
has been greatly preoccupied with its legitimacy ever since, and consequently, 
to legitimise itself and stay in power, maintaining economic growth, social sta- 
bility, and national unity have been the party-state’s top priorities (Laliberté & 
Lanteigne 2008; Zhu 2011). As Chapter 4 illustrates in more detail, a cautious 
stance towards foreign influence remains perhaps the most enduring legacy of 
the unrest of 1989, because it is seen as a major contributor to the unrest. 

Although economic growth initially slowed after 1989, education reform to 
maintain it has only intensified since the 1990s (Postiglione 2011). As the basic 
quantitative targets had largely been achieved by then, reflecting a larger soci- 
etal trend whereby all human resource decisions were justified in terms of qual- 
ity, emphasis in education increasingly shifted from quantitative to qualitative 
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improvement (Kipnis 2006; Law 2007; Liu 2008). Following this shift in focus, 
and largely inspired by academics now familiar with Western ideas and prac- 
tice, the notion of “quality education” (suzhi jiaoyu) appeared in the education 
agenda. The notion gained momentum in the 1990s as a guiding principle 
of education policy, and it soon became a catchphrase for Chinese policy- 
makers, the media, and the general public (Chen Cravens, Chu & Zhao 2011; 
Della-lacovo 2009; Kipnis 2006). Since its first appearance in the mid-1980s, 
the wording has been used in policy resolutions and reform plans, and in the 
revised (2006) version of the 1986 Compulsory Education Law, quality edu- 
cation was promoted from a policy to the legal level for the first time (Della- 
Iacovo 2009; Law 2007; Liu 2008). Consequently, new policy orientations place 
more emphasis on efficiency, practicality, plurality, students’ all-round devel- 
opment, and the measurement of student and school outcomes, although the 
ultimate goal of quality education is still to assist the party-state in its quest for 
national rejuvenation (Chen Cravens, Chu & Zhao 2011; Han & Yang 2001; 
Della-lacovo 2009, Xin & Kang 2012). As Belinda Della-Iacovo (2009: p. 242) 
explains, 


The ultimate goal of suzhi jiaoyu is national strength. The government has 
identified the skills China’s workforce needs to acquire which it sees as 
critical to sustaining its modernisation drive. Practical skills and innovative 
ability have been identified by the leadership as areas which need to be 
improved in order to raise China’s global competitiveness. 


In addition to this qualitative turn, other notable changes also took place in the 
Chinese education field around the same time. First, following a broader shift 
in Chinese society towards market practices, the education system also became 
increasingly market-driven as a result of a policy known as “entrepreneurial- 
ising education” (jiaoyu chanyehua), which emerged in the late-1990s (Ngok 
2007; Qi 2011). As Gerard A. Postiglione (2011: p. 85) asserts, “Markets have 
come to matter more than Marxism in educational provision, especially as more 
needed to be paid for quality education”. Second, by the early 2000s, the idea of 
a “scientific outlook on development” (kexue fazhan guan) was gaining ground 
in the political leadership. These ideas consolidated the notion that policymak- 
ing in the field of education should also be based on evidence and science 
(Chen 2010; Zhang 2010; Xu & Li 2011). 

With the change in the definition of quality in education, new measures 
to ascertain that it is achieved have been required. Consequently, the reform 
of the testing, evaluation, and assessment system has played a central role in 
promoting and improving the quality of the education system (Han & Yang 
2001; Liu 2008; Peng, Thomas, Yang & Li 2006; Xu & Li 2011). Most notably, 
this change has manifested itself in the construction of the assessment (jiance) 
system headed by an institution known as the National Assessment of Educa- 
tion Quality (NAEQ, Jiaoyubu Jichu Jiaoyu Zhiliang Jiance Zhongxin), which was 
piloted in 2007. This formally independent system managed by the NAEQ 
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runs in parallel with the supervision system by operating on the four already 
mentioned levels. However, its main purpose differs from that of supervision. 
The jiance system is tasked with gathering data through standardised tests, which 
are then utilised in policymaking and system development (Chen 2010; Fan & 
Liao 2013; Wang & Zhang 2009; Zhang 2010; Zhou 2012). Rhetorically at 
least, this assessment plays a central role in contemporary Chinese compulsory 
education. We examine the system’s various aspects, as well as its disparity 
between rhetoric and practice, in more detail in Chapters 4 and 6. Several 
other reforms, such as curriculum reform, which touch on QAE have also been 
initiated. 

At the beginning of the twenty-first century, the People’s Republic of China 
is arguably both politically and economically a regional and global powerhouse. 
China’s share of world gross domestic product has risen to double digits, and 
after four decades of economic reform, the Chinese population has become 
increasingly rich. Access to all levels of education has grown exponentially, and 
illiteracy among young people has been virtually abolished. Many problems 
persist, however. Despite the economic progress of the last forty years, politi- 
cal reform has lagged. No change has been made to the fundamental structure 
of the political system since the events of 1989, and the ultimate authority 
to make policy still resides with the Chinese Communist Party. This is also 
evident in the field of education, where regulations and overall plans are still 
largely formulated by the Ministry of Education, whereas sub-national authori- 
ties are mainly responsible for implementing policies by adapting to local con- 
ditions (which sometimes entails risking the circumvention of central policies) 
(Postiglione 2011; Qi 2011). Although some flexibility and innovativeness are 
allowed, all local adaptations must be done within the general framework, and 
it is primarily the workload — not the authority — which has been transferred 
downwards (Harris, Zhao, & Caldwell 2009; Ngok 2007; Qi 2011). Tingting 
Qi (2011: p. 34) has aptly described this phenomenon as “centralised decen- 
tralisation”. Although the fundamentals of the system remain unchanged, this 
does not mean everything has remained the same. Jessica C. Teets (2013) has 
noted that a new model of state-society relationship, which she calls consul- 
tative authoritarianism, is emerging in Chinese society at large. She suggests 
(ibid.: p. 32) this reformed form of governance 


merges the regulatory-state idea of a relatively autonomous civil society 
collaborating with the state to solve social problems with the New Left 
idea of state guidance of groups to protect society from narrow inter- 
est groups and social instability. This model is characterized by two main 
aspects — a pluralistic society participating in policy formation and imple- 
mentation, and the use of multiple indirect tools of state control. 


As we shall see in Chapter 4, these tendencies have also been evident in the 
structure of the current QAE system in Chinese compulsory education as new 
stakeholders in the field have emerged. The continued emphasis on testing and 
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examination performance, which dates to imperial times, still largely structures 
systems (Postiglione 2011). As our interviews also indicated, the influence of 
these practices is so strong and pervasive that wordings such as promotion- 
oriented education (shengxue jiaoyu) or examination-oriented education (ying- 
shi jiaoyu) are today staple features of the vocabulary of Chinese educational 
discourse: the perceived negative influences of these continuing practices were 
frequently mentioned by our interviewees. Resource issues also persist. Many 
rural schools still lack qualified staff and facilities (Robinson & Yi 2008). Indeed, 
the economy's exponential growth has given rise to new types of inequality. 
There is not only a gap between different areas and provinces today, but the 
gap between counties and even schools — of which a growing number are now 
privately run — in the same locality is continually growing (Gustafsson, Shi, & 
Sicular 2008; Huo & Gao 2006). 


Russia: quality control reinforcing central regulation' 


School education in Russia has been facilitating state development projects 
since the tsarist era. Two distinct societies and cultures, the elite and the peas- 
antry, co-existed in imperial Russia, and by the beginning of the twentieth 
century, teachers in peasant schools were expected not only to spread liter- 
acy but also to play the role of political mediators, who would assist in the 
implementation of the state’s political and economic strategy for Russia’s vast 
rural community (Seregny 1993: pp. 121-122). In Soviet Russia, education 
was of major importance to the effort to build a society of egalitarianism and 
solidarity. Schools were to equip students with everything needed for their 
effective future contribution to the socialist economy and to the project of 
building communism. Hence, alongside the development of students’ skills and 
knowledge, schools were involved in ideological indoctrination. The Stalinist 
school model was developed in the 1930s after a decade of pedagogical experi- 
ments, and it reinstated many aspects of pre-revolutionary education, such as 
the teacher’s dominance of the classroom and strict adherence to a centralised 
curriculum (Byford & Jones 2006: p. 422). The main characteristics of the 
Stalinist school were rigid discipline and hierarchy, the enormous quantity of 
information students were expected to absorb, and an emphasis on collectivism 
and patriotism. The state assumed full responsibility for guaranteeing free and 
equal access to education, as well as its quality. At the same time, it was the duty 
of every (future) citizen to attend school and to study to the best of his or her 
ability to develop into an efficient member of socialist society (Livschiz 2006: 
p. 559). Students’ commitment to learning was overseen by school workers, 
student organisations in which every student was enrolled from grade one, and 
local organs of the Communist Party, which exerted influence over parents at 
their workplaces. 

The main instruments of quality control during the Soviet era were inspec- 
tions, school reports, and the school census. In 1944, a system of grades and 
examinations was established. Students’ outstanding results were symbolically 
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rewarded with gold and silver medals. There was no centralised national exami- 
nation, and school-leavers” educational achievements were assessed by schools 
under the supervision of the local education authorities. It was assumed that a 
centralised curriculum and teacher training, given that schools were provided 
with sufficient resources, effectively ensured a high standard of education and 
its relevance to the current needs of the state (West & Crighton 1999). Until 
the education reforms of the 2000s, the grades of students and their related 
indicators, such as the percentage of students successfully transitioning to the 
next school year or the numbers of graduates awarded medals, served as the 
main measures of quality (see e.g., Bakker 1999: p. 296). Assessment of schools 
and teachers on the basis of students’ grades was prohibited by the decrees of 
the 1940s, and competition in socialist education was discouraged (Mayofis 
2015: pp. 40-41). Regular school inspections ensured teachers’ objectivity in 
their assigning of grades: 


The mark ... should reflect the true knowledge of students. When fighting 

the poor progress of students, one should not follow the path of lowering 

the standards, as some teachers tend to do. Only through raising the stand- 
ards the quality of knowledge can be improved. 

(from an article in the Teacher’s Newspaper, 1948, 

quoted in Mayofis 2015: p. 82) 


Despite the appearance of state control over every aspect of school life and 
the constant improvement in quality, Soviet education faced many challenges. 
The country’s vast size, its social and ethnic diversity, and the enormous differ- 
ences between the Soviet city and countryside contributed to the large variety 
of implementation methods for centrally designed education policies. In the 
aftermath of the economic and social devastation of collectivisation and World 
War II, schools were severely underfunded, and post-war state statistics regis- 
tered massive dropouts and repeating of study years (Livschiz 2006). Teachers 
were blamed for this: they were accused of “formalistic” teaching approaches 
and lacking the necessary pedagogical skills (Mayofis 2015: pp. 61-64). Between 
1940 and 1956, schools charged tuition fees for grades 8—10, making educa- 
tion at this and higher levels virtually unaffordable to poorer families, especially 
in rural areas. Studies of Soviet education consistently point to its persistent 
inequality (Byford & Jones 2006). 

Ideas about the mission of education in Russia started to change under pere- 
stroika in the 1980s. The key idea of perestroika was to reorganise the social 
and political sphere to allow for more freedom and truth. In education, this was 
interpreted as a call for more democratic and student-centred schools. A group 
of educators within the Academy of Pedagogical Sciences, called VNIK “Bazo- 
vaia shkola”, proposed a critical reform of the Soviet school. They suggested 
a set of new basic principles: democratisation, humanisation, differentiation, 
openness, continuity, and a developmental approach to education. They also 
promoted greater participation by school staff in a wide range of decisions 
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concerning schoolwork, claiming that “educational processes should be gov- 
erned by those who teach” (Long & Long 1999: p. 88). School administration 
was envisioned as a communal enterprise which should involve students and 
parents, as well as teachers and civil society representatives, to better serve local 
needs. Another phenomenon of the 1980s which facilitated education reform 
was the movement of “innovative teachers” (pedagogi-novatory). These teachers 
offered creative teaching methodologies with a focus on students” individual 
abilities and interests, demonstrating that this greatly increased students” learn- 
ing. In 1986, they formulated the proclamation of “Cooperative Pedagogy”, 
based on partnership between teacher and student. The concepts and meth- 
odologies of innovative teachers were disseminated in courses, seminars, and 
media sources. 

The ideas advocated by VNIK and innovative teachers became the basis of 
the new Law on Education in 1992, following the dissolution of the Soviet 
Union. The leader of VNIK, Eduard Dneprov, became the first minister of 
education in post-Soviet Russia. The new law also enabled the decentralisation 
of education to support school-level management, allow schools to tailor the 
curriculum to local needs, and facilitate school choice for families. The govern- 
ment’s administrative and fiscal responsibilities were shifted to regional and local 
education authorities. However, the call for democratisation, humanisation, and 
differentiation was not the only factor in the dramatic changes in Russian edu- 
cation. The economic crisis of the early 1990s caused severe underfunding. The 
abrupt transition to a market economy led to the reinterpretation of education 
as a service and the partial privatisation and commercialisation of the school 
sector. Alongside the legislation for decentralisation, inequality in education 
sharply increased (Polyzoi & Dneprov 2011). 

While the central authorities lacked financial resources, new education ini- 
tiatives were financially supported by international philanthropic foundations, 
especially the Soros Foundation (Startsev 2012). Other international organisa- 
tions, including the World Bank and the OECD, started working in Russia in 
the 1990s. They encouraged efforts to democratise and differentiate education 
and suggested the implementation of a more thorough and modern system 
of quality assurance, which would include nationwide quality standards and 
external quality evaluation mechanisms and involve diverse stakeholders in the 
QAE process (OECD 1998; World Bank 1999). World Bank experts regarded 
the lack of standardised achievement testing providing commensurable national 
statistics as an absence of QAE policy (World Bank 1995). Both the World 
Bank and the OECD recommended standardised testing as a key quality evalu- 
ation mechanism. At the same time Russia started participating in international 
large-scale assessments, TIMSS (Trends in International Mathematics and Sci- 
ence Study) was undertaken in 1995 and 1999, and Progress in International 
Reading Literacy Study (PIRLS) and PISA were added in the 2000s. The 
methodology of these tests and their sociological perspective served as a source 
of inspiration for Russian scholars, who conducted meta-analyses of test results 
and designed similar national instruments (Piattoeva & Gurova 2018). 
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Ideas about education quality and its evaluation changed in line with the 
developments of the 1990s. Quality came to be understood as intrinsically 
diverse, based on “customer” needs and context-bound. Assessments were 
addressed primarily as a tool for school and classroom internal diagnostics 
with the overarching aims of the effective management of schools and meeting 
the needs of students and parents. At the same time, the literature on school 
management published in this period promoted an evidence-based and goal- 
oriented approach and insisted that education outcomes, although specific to 
each school, should be measurably defined and accompanied by a set of indica- 
tors (e.g., Tret'iakov 1997). The development of measurements in education was 
also powered by an increasing interest in ability testing, which had been banned 
during the Soviet era, and the rise of information technologies, which opened 
new possibilities of collecting and processing large quantities of data. While 
these new ideas were not reflected in the education legislation of the 1990s, they 
arguably prepared the ground for the changes of the following decade. 

At the beginning of the 2000s, with the recovery and growth of the Rus- 
sian economy and change in political leadership, new education reforms 
were launched. The Ministry of Education issued several key documents: the 
National Doctrine for Education (2000), the Concept of Modernization of 
Russian Education (2001), and the Federal Strategic Program for the Develop- 
ment of Education (2005). These documents defined the mission of education 
primarily in economic terms: education should contribute to socio-economic 
growth, serve the needs of the labour market, promote innovation, and ensure 
the global competitiveness of the Russian economy. They supported the fur- 
ther introduction of market mechanisms to the education sector and called for 
efficiency, accountability, and transparency in education. These ideas accorded 
with the recommendations of international organisations and were in line with 
general performance management reforms in Russia (Gounko & Smale 2007; 
Gusarova & Ovchinnikova 2014). In 2001, the World Bank started its Educa- 
tion Reform Project in Russia, aiming to “improve quality and standards; pro- 
mote the efficient and equitable use of scarce public resources for education; 
modernize the education system”. The project was launched at the federal level 
and piloted in three Russian regions. Its intention was to introduce quality 
monitoring and statistics, based on a new system of assessing student outcomes 
in the governance of general education (World Bank 2001). 

Several major instruments of quality assurance and evaluation were intro- 
duced in the educational reforms of the 2000s. The Ministry of Education 
outlined “state standards”, to which school curricula were obliged to conform. 
New procedures for the licensing, attestation, and accreditation of educational 
institutions were developed, ensuring schools and universities complied with 
central regulations and produced satisfactory results. In 2001, the national 
school-leaving examination (the Unified State Exam, USE, or GIA-11) was 
piloted in several regions, and in 2009, it became compulsory for all school- 
leavers. The examination combined the functions of the school graduation 
test, the national university entrance test, and the source of national statistics 
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on educational achievement (see Chapter 8 for more details on the USE). The 
Unified State Exam was claimed to facilitate fairer access to universities by 
assessing school-leavers objectively and impartially and to improve the quality 
of education across the country, fostering compliance with the official curricula. 
Thus, the USE was designed to contribute to greater equality for school-leavers 
and address the negative effects of the radical decentralisation of the 1990s. It 
also served as an important source of objective information for evidence-based 
policymaking, which was perceived as indispensable for modern regulatory 
methods. The national academic discussion between 2000 and 2010 also shifted 
its focus from school-based, customised evaluation of a largely qualitative nature 
to depersonalised, often numerical data on educational achievement generated 
by national and international assessments. 

The latest State Program for Education Development (for the years 2013 
to 2020) includes a programme for “the development of the system of quality 
evaluation of education and transparency of the education system” as one of its 
main elements. The national QAE system outlined in this document comprises 
action for the state regulation of education, assessment of educational achieve- 
ment (the national examinations (GIA) after grades 9 and 11), procedures for 
independent quality evaluation, and the participation of Russia in international 
studies (Government of Russia 2012: p. 218). Quality of education is defined 
primarily as compliance with state standards and high performance in inter- 
national tests. Scores in national examinations serve as the main indicators of 
education quality at all governance levels, although the examinations’ contents, 
procedures, and administrative uses continue to be discussed and altered. Dur- 
ing the last decade all-Russia measurements of educational achievement have 
been added to the QAE system to complement national examinations and 
international studies. 

The introduction of QAE instruments facilitated the restoration of centralisa- 
tion in Russian education. Evaluation procedures and associated performance- 
based incentives were added to the traditional instruments of quality control, 
instead of replacing them. Despite appearing to be piloted, national exami- 
nations were implemented consecutively in all regions of the country in an 
authoritarian manner. The state defines quality criteria and controls as both 
inputs (through institutional accreditation and audit) and outputs (through 
measurable outcomes) of general education. Researchers argue that Russian 
education reforms can be characterised as “authoritarian modernisation” and 
that a neoconservative turn is taking place, legitimised by a neoliberal, appar- 
ently Western, rhetoric (Minina 2016). At the same time, the current school 
system shows significantly more diversity and inequality than in the Soviet 
era, while national regulations still allow for considerable local variation in 
implementation. Contemporary education governance in Russia, including 
QAE instruments, evinces a mixture of Soviet-era legacies, developments from 
the democratisation and diversification period of the 1990s, and neoliberal 
influences. 
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Identifying commonalities and 
differences in paths to QAE 


As our work indicates, Brazil, China, and Russia have followed — and to a large 
degree continue to follow — very different developmental paths in their socio- 
political contexts and in the place of education within the larger contexts of 
politics and governance. However, towards the end of the twentieth century, the 
countries’ paths intersected as they were shaped by a shared interest in QAE. 
In each country, a contingent conjuncture can be identified in which different 
developments converged to shape this interest. 

In Brazil, this conjuncture dates to the late 1980s. QAE practices emerged at 
the same time as massive school expansion, deepening federalism and decen- 
tralisation, increased democratisation, public managerial reforms, and coopera- 
tion with international organisations. Unplanned development of school access 
and significant dropouts and retention rates presented a problem, while the 
democratisation of the country and the consequent political acknowledgement 
of education as a social right provided a legitimate political framework. The 
ongoing managerial reforms and experiments with assessments funded by the 
World Bank and supported by the UNDP offered a solution. The intersec- 
tion of these contingent developments formed the breaking point which set in 
motion the implementation of the Brazilian QAE system, which was largely 
composed of large-scale assessments and quantitative indicators. QAE is viewed 
ambivalently in the education arena because, although it is criticised by many, 
it is also recognised as an instrument which can tackle the disparities and ine- 
qualities which, despite much progress in education, are still deeply rooted in 
Brazilian society. 

In China, the conjuncture can be traced roughly to the period between the 
mid-1990s and the early 2000s. First, with the help of established institutions 
like examinations and the re-established supervision system, the quantitative 
targets of the “Two Basics” had been largely achieved by then. Second, main- 
taining economic growth had begun to play an increasingly important role in 
the party-state’s drive to enhance its legitimacy after 1989, while the reform 
of the education system was crucial to the creation of economic growth. Third, 
the opening of Chinese society to the world brought new ideas. On the one 
hand, a growing number of Chinese had become familiar with Western educa- 
tion practice and, as Chapter 4 elaborates, international organisations were also 
increasingly active in the education field. The notion of quality education — 
suzhi jiaoyu — rose to the national agenda. On the other, ideas about the mar- 
ketisation of education and the idea of evidence-based policy were also gaining 
ground. These new ideas changed how quality in education was defined, which 
in turn directly contributed to the reforms of the QAE system — most notably, 
the establishment of the jiance system — as new mechanisms to ensure quality 
were required. However, these processes are by no means complete. Chinese 
society and education face new and continuing problems. 
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In Russia, the conjuncture occurred at the end of the 1990s, when national 
movements for greater diversity and student-centred education, political con- 
cerns of increasing inequality and decentralisation, and international examples 
of and recommendations for large-scale assessments of student achievement 
converged. Humanisation and democratisation, as part of the de-Sovietisation 
of schools, led to a focus on student learning, a diversification of curricula 
and teaching methods, and the accountability of the school to a wide range 
of stakeholders. Education inequality increased dramatically because of severe 
underfunding, the marketisation of education, and a widening of social and 
economic gaps between families. By the end of the 1990s, the introduction of 
regional autonomy and transfer of administrative and fiscal responsibilities to 
local authorities had resulted in widely varying conditions and education prac- 
tices in different regions. These factors contributed to what the public perceived 
as an education crisis, and justified the re-instatement of central control over 
education quality and the quality assurance of education opportunities, both 
of which had been long-standing prerogatives and social obligations of central 
government. Participation in international student achievement assessments 
also contributed to the development of national measurement instruments, and 
advice from the World Bank and OECD promoted modern regulatory instru- 
ments, especially standardised testing, as a source of evidence for policymaking. 
These developments in the early 2000s resulted in the establishment of a new 
QAE system based on state standards and national school-leaving examinations. 
The education system was recentralised through new regulation and under the 
motto “equal access to quality education”. 

From increased democratisation in Brazil to concern about regime legiti- 
macy in China and de-Sovietisation in Russia, to name only a few examples, 
these developmental paths clearly reflected the unique cultures and political 
systems of the countries. QAE arose as a response to markedly different prob- 
lems. However, these paths share some distinctive similarities, some of which 
are evident in all cases. First, in all cases, increased international influence and 
cooperation played a striking role in the introduction of QAE. Second, in all 
cases, the marketisation of education and public managerial reforms were an 
important constitutive factor. Some similarities were either identifiable or more 
evident in two cases. Although not a factor contributing to interest in QAE per 
se, it is noteworthy, first, that this interest in QAE did not emerge entirely simul- 
taneously: in Brazil, changes were already evident in the late 1980s, whereas 
they followed in China and Russia a decade later. Second, China and Russia 
also resembled each other in the sense that the change at this time towards more 
student-centred education thinking played a pivotal role in both. Third, the 
massive education expansion of the previous decades was especially important 
in Brazil and China. Finally, and most obviously in Brazil and Russia, anxiety 
about the decentralisation of the education system was also an important factor 
in growing interest in QAE. 

Paths converged only temporarily, as this and the following chapters show. 
Not only had shared interest in QAE arisen in response to markedly different 
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challenges and problems, but also, once operationalised, QAE reforms reflected 
the obstacles and limitations set by the very different cultural and political 
settings of the respective countries. Consequently, as the following chapters 
elucidate, since they were operationalised, QAE reforms in Brazil, China, and 
Russia have again diverged. 


From norm takers to norm makers 
in the global environment? 


While all this was happening within the countries, they had also begun to 
engage increasingly with the international community. Brazil had been engag- 
ing with international organisations since World War II, and by the 1990s, 
China and Russia were also involved (see Chapter 4). By 2017, all were mem- 
bers of the most prominent and influential global governance institutions, for 
example, the International Monetary Fund (IMF), the United Nations, the 
World Trade Organisation, and the World Bank. To borrow the words of Pu 
(2012), the countries had gradually become socialised within the international 
order as responsible norm takers. Since the turn of the millennium, however, 
there have been signs that Brazil, China, and Russia are no longer content sim- 
ply to engage with the international community and follow its rules. There are 
signs that socialisation has become a two-way process, as emerging countries 
aspire to become norm makers in the international order (Pu 2012). Part of 
this drive has been the construction of a network of global governance institu- 
tions which in part complement and in part compete with the current US and 
Western-dominated system. With three former Soviet Republics, China and 
Russia founded the Shanghai Cooperation Organisation in 2001 to further 
their political, economic, and military cooperation. In 2016, the Asian Infra- 
structure Investment Bank was launched as a Chinese-sponsored alternative to 
the World Bank, and in South America, Brazil played a central role in the estab- 
lishment of the Union of South American Nations. These initiatives belong to 
the larger global context of increasingly numerous and influential emerging 
power alliances, which are not limited to the activities of Brazil, China, and 
Russia (Bava 2011). 

Within this wider context, perhaps most notably and interestingly because of 
their shared membership, the countries are also cooperating under the BRICS 
framework. In 2001, the acronym BRIC was used for the first time to group 
Brazil, Russia, India, and China, and five years later, a political dialogue between 
the countries began. The acronym BRICS was coined in 2010 with the inclu- 
sion of South Africa (Keukeleire & Hooijmaaijers 2014; Pant 2013). Initially, 
as Luckhurst (2013: p. 252) states, the countries were lumped together because 
of “the dramatic economic growth predictions for the original four BRICs; 
the likely importance of their future governance role in the world economy; 
and the nature or quality of the development of these countries”. Since the 
genie escaped the bottle, justifications for grouping these countries together 
have been numerous, ranging from their developmental requirements to their 
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military stances. From the outset, however, academics, journalists, and politi- 
cians have been divided on the grouping's nature, justification, and prospects 
(Briitsch & Papa 2013). 

It is obvious, however, that the countries themselves have exploited percep- 
tions of imminent commonality. The global financial crisis which began in 
2007-2008 provided them with the ideal opportunity to develop common 
positions highlighting the contrast between a prospering periphery and a strug- 
gling core. The historic IMF quota reforms and the threat of a joint walkout 
from the 2009 United Nations Climate Change Conference if their common 
minimum position was not accepted served as ideal opportunities to symboli- 
cally highlight the “rise of the rest” narrative (Liu 2016; Pu 2012; Stuenkel 
2014; Unay 2013). 

The BRICS remain a somewhat informal cooperation platform. However, 
China and Russia seem especially eager to push their cooperation in a more 
formal direction (Liu 2016). The establishment of the New Development Bank 
in the summer of 2014 as an alternative to the World Bank and the IMF and 
the launch of the “eThekwini Action Plan” to address issues related to for- 
eign policy cooperation mark the first steps on this road (Keukeleire & Hooi- 
jmaaijers 2014; Liu 2016; Stuenkel 2014). Those who question the validity of 
grouping the countries based on their similarities are for the same reason also 
sceptical, however, of the prospects for their effort to strengthen cooperation 
and further institutionalise the BRICS (see, for example, Liu 2016). 

Although the BRICS grouping has largely been seen (and has portrayed 
itself) as politico-economic, there are signs that the countries have begun to 
cooperate in other areas under the BRICS umbrella — including education. 
Russia and China, for example, have initiated a platform for higher educa- 
tion cooperation which has been described as aiming “to become a platform 
for academic and expert cooperation, comparative research, and international 
educational projects” (Higher School of Economics 2013). Most important, 
perhaps, the BRICS countries’ ministers of education meet regularly to pro- 
mote education cooperation. At their autumn 2016 meeting, the New Delhi 
Declaration on Education was unveiled to promote cooperation in education. 
The declaration included — at least rhetorically — areas of cooperation such as 
“quality education” and “quality assurance” (The Indian Express, 30 Septem- 
ber 2016; Xinhua, 30 September 2016). 

Research focusing explicitly on the link between education and the BRICS 
countries has been scarce, however, and has so far been concerned exclusively with 
higher education (see, for example, Altbach & Bassett 2014; Carnoy et al. 2013; 
Schwartzman, Pinheiro, & Pillay 2015). Although we do not explore this further 
ourselves, we hope our contribution paves the way to such new research ventures. 


Note 


1 This section draws on earlier research articles, namely Gurova (2017) and Gurova, Piat- 
toeva, and Takala (2015). 
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Introduction 


Our book’s key questions concern how agendas and actors’ arenas interact. 
In analysing the policy process, Baumgartner and Jones observe that politi- 
cal change happens in ruptures after the agenda is changed by growing posi- 
tive, change-supporting feedback, such as diffusion from other political systems, 
which catalyses change in the political sub-system’s arrangement (Baum- 
gartner & Jones 2009). An analysis of this in relation to quality assurance and 
evaluation (QAE) is of interest to our research. 

We can identify a transnational QAE agenda. As we pointed out in Chap- 
ter 1, instead of looking at all the contents of this QAE “megatrend” (Vedung 
2003: p. 1), we focus on how the integration of global education policy dis- 
courses and practices and the general interest in measuring quality in education 
have changed and reshaped the actor relations within a polity. For example, 
there is evidence of the effects of QAE on the roles of national education policy 
actors in how information and data have provided the means for an attempt 
to regain central power (Ozga et al. 2011) while simultaneously, and somewhat 
paradoxically, creating an image of a more decentralised, horizontal, and net- 
worked understanding of knowledge (Ozga 2012). To reiterate Chapter 1’s 
idea, it is only when a slippery and undefinable notion of quality is applied that 
the QAE agenda starts to affect the national political arena. For example, new 
organisations responsible for large-scale assessments are introduced, leading to a 
rearranging of the existing organisations’ roles. 

When a rupture of the status quo happens, “[t]he old policymaking institu- 
tions find themselves replaced or in competition with new bodies that favor 
different policy proposals” (Baumgartner & Jones 2009: p. 4). Central to the 
understanding of these dynamics is the analysis of sub-systems such as the edu- 
cation system and the possible QAE expert communities which exist within 
it. Baumgartner and Jones discuss “policy monopolies” which have an institu- 
tional structure connected to policymaking and exclusive access to expertise 
and which legitimise their action with the help of core political values such as 
fairness or economic growth. These monopolies can be disturbed, for exam- 
ple, with the help of agenda-setting through the media, whereas apathy helps 
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to sustain a monopoly (Baumgartner & Jones 2009). We assume that QAE 
expert communities meet this definition. QAE policy meets this characterisa- 
tion in terms of exclusion through expertise and the core political values of 
enhancing learning and economic growth. In our research, we are interested in 
understanding the extent to which the potential positive, change-supporting 
feedback on the agendas of these three countries has resulted in a change in 
polity. 

Using the CADEP terminology adopted in Chapter 1, this chapter starts 
an analysis of the political constellations of actor, that is, the political situation. 
We demonstrate that the political constellation of actors is inseparable from 
changes in the second dimension of CADEP, namely the political possibilities. 
In attempting to understand the relationship between these two dimensions, we 
use the idea of Baumgartner and Jones (2009), who claim that policy agendas 
and the structure of the political system are interconnected. Distinguishing the 
relevant actors is also connected with the idea of what is on the agenda. Our 
primary focus is on structural change, but we also link this to the changing 
QAE agenda. We aim to analyse the changing national political situation in 
Brazil, China, and Russia in relation to the possibilities and restrictions created 
by the effort to measure and improve the quality of school education. 

Using the data presented in Chapter 2, this chapter focuses on documentary 
and interview data from individual actors — the international and key national 
organisations involved in QAE policy and practice in the three countries. We 
focus on interviews at international and national levels. These interviews and 
documents largely sourced online limit our empirical interest to developments 
since the turn of the millennium. 

Although all three countries have a history of suspicion to various degrees 
towards international actors, since at least the 1990s, they have become increas- 
ingly involved in international cooperation in the field of education quality 
and its evaluation, which means that many actor relations and agendas are also 
transnationally linked. To understand the changing constellations of actors in 
QAE policy, our analysis focuses on two dynamics. The first relates to how the 
development of expert communities is networked transnationally to interna- 
tional actors, the second to how the changing polity has activated other actors. 


Building national QAE education policy communities 


The development of QAE capacity and tools in Brazil, China, and Russia is 
also connected to external advice from international organisations. However, 
the timing and extension has varied in each case. In line with our theory, the 
technical side of large-scale assessment is a reflection of actor relations. As we 
pointed out in the book’s introduction, the question of handling, designing, and 
planning data use is political, and it therefore makes sense for national govern- 
ments to develop their capacity. Where the interconnectedness of agendas with 
actor relations is concerned, we observe that agendas slowly began to adopt 
QAE ideas in the 1990s and that its feedback resulted in different degrees 
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of polity reorganisation. In each context, the development of national expert 
communities is tightly linked to financial resources and political decisions. 


The changing agenda in the 1990s 


During the 1990s, in all countries, the international and transnational influ- 
ences of QAE began to make an impact. The international community affected 
this in two ways: through direct recommendations from international organisa- 
tions, especially the World Bank, and through the influence of the internation- 
ally networked expert community. In each context, the mix varied. 

In Brazil, there was a gradual accumulation of national expert networks and 
their capacity in parallel with international organisations’ involvement. Brazil 
differs from Russia and China in its longer-term cooperation with the Organ- 
isation of American States, the US American Aid Institution (USAID), the 
United Nations Educational, Scientific and Cultural Organisation (UNESCO), 
and the World Bank, among others, which has been a reality since World 
War II. A research article from our project concluded: 


Education-indicator programmes have blossomed since the 1990s, along- 
side global education-related IO [international organisation] programmes. 
Independently of their political position, all the post-dictatorship govern- 
ments took steps towards implementing assessment with a view to improv- 
ing quality in education, a strategy that mirrored global developments. In 
this development, we have pointed out the importance of the long build- 
up of QAE capacity and webs of experts. 

(Kauko et al. 2016) 


China’s engagement with the international community began in earnest only 
in the post-Mao period, and it was not until the 1980s that China joined the 
entire network of intergovernmental organisations (IGOs) (Kent 2002; Ngok & 
Kwong 2003). By the 1990s, China’s policy of opening to the world had 
increasingly exposed the Chinese education community to new foreign ideas, 
and international education issues had noticeably surfaced. Journals increasingly 
dealt with the subject, and the first government research units were established 
(Altbach 1991). Various IGOs, such as UNESCO, through its global Education 
for All movement, the World Bank, the OECD, and the European Union also 
became increasingly active in promoting transnational education practices and 
programmes in China (Ngok & Kwong 2003; Suominen et al. 2017). By the 
mid-1990s, many international non-governmental organisations (INGOs) had 
also begun to launch projects and establish offices in the country (Ma 2002; 
Deng 2010; Yang G. 2005). 

As Chapter 3 described, simultaneously, and partly resulting from these 
international contacts, the Chinese education system had also begun to shift 
from quantitative to qualitative targets to boost the state’s modernisation 
drive. Many reformers who advocated these ideas, later known in the Chinese 
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discourse as “quality education”, were familiar with Western education theory, 
and the North American notion of “competency-based education”, empha- 
sising specific competencies instead of exam success, was especially popular 
(Kipnis 2006). 

This shift in priorities also required the development of a national QAE 
capacity. International experience and practice were especially influential in the 
construction of the national assessment system (Zhang & Wan 2017). An inter- 
viewee explained, “As a matter of fact, it [the establishment of the assessment 
system] is also a reply to the demands of globalisation. So, against a backdrop 
like this, we also wanted to understand what the current situation of Chinese 
education was like” (CN-E-02). The OECD’s Programme for International 
Student Assessment (PISA) surveys’ strong influence on the development of 
this national assessment system is acknowledged by practitioners and academics 
alike (the assessment has even been called the “Chinese PISA” — a formulation 
used by many of our interviewees), although direct borrowing has occurred 
mainly in technical areas. 

In post-Soviet Russia, international organisations gained some prominence 
between the mid-1990s and mid-2000s. In this period’s early years, in a bilat- 
eral Dutch-Russian project, the World Bank and the OECD prepared reports 
on Russian education. These reports were diplomatic enough to give credit to 
the achievements of the Russian education system during the Soviet period, 
but the basic message was one of strong criticism of the traditional Russian 
notion of what constituted quality education (Leonteva, Plomp, & Voogt 1998; 
World Bank 1995; OECD 1998). This pedagogical tradition was criticised as 
unsuitable for the needs of a market economy, and a new system of assessment 
was recommended as a tool for changing the mode of learning, examinations, 
incentivisation, and information gathering. As evidence for this argument, ref- 
erence was made to the low performance of Russian students in PISA, in sharp 
contrast with their excellent performance in the Trends in International Math- 
ematics and Science Study (TIMSS) and Progress in International Reading 
Literacy Study (PIRLS) tests. However, these recommendations resonated with 
the views of a reform-minded faction of Russian policymakers and education 
experts, and it is impossible to demonstrate a causal effect between the two. Yet 
very few QAE reforms took off at the national level during this period, apart 
from the piloting of the national school-leaving examination. 

In the late 1990s and early 2000s, most changes in Russia were first imple- 
mented at the sub-national level, channelled through World Bank-funded 
project work in selected pilot regions. During this period, many assessment 
consultants travelled to Russia to promote their views on appropriate policy 
and practice, typically with very little knowledge and understanding of the 
existing baseline situation. Our interviews indicate that Russian experts con- 
sidered much of the advice patronising and inflexible. Nevertheless, after this 
troubled start, gradual processes of dialogue followed. In the Republic of Chu- 
vashia, where we conducted our fieldwork, the World Bank-funded project 
(operational between 2002 and 2006) was generally regarded by our inter- 
viewees as having produced lasting results in capacity building, and the same is 
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acknowledged of the bank's Project Completion Report (World Bank 2007). 
One of our interviewees, who has held a prominent administrative position, 
described this process as follows: 


Everything was new for us, so our views did not mean much, while they 
[the international experts] had seen a lot, and there were a lot of compro- 
mises in the discussion and defence phases [of the proposed projects] ... 
Sometimes it [discussion on the intermediary reports] got really crazy. 
Either the experts or our people would cry because of the misunderstand- 
ings or tough approach to a position if you couldn* prove it, even if you 
were absolutely convinced you were right. So, they were very rigorous 
discussions, very deep ones. 

(RU-E-15) 


This aptly describes the general influence in all cases: data expertise is an appeal- 
ing argument for national steering and guidance. Each of the countries’ expert 
networks were engaged in dialogue. 

The agenda in each country therefore evolved during the 1990s to support 
the idea of QAE in education governance. This began to bear fruit in conjunc- 
tion with the changing financial and political situation during the new millen- 
nium and eventually affected the political constellation of actors. 


Building capacity 


The extent to which Brazil, China, and Russia have worked with international 
organisations varies, and the nature of cooperation seems to depend on their 
respective financial and political situations. 

At the turn of the twenty-first century the World Bank became more active 
in Brazil (Kauko et al. 2016). However, its focus has shifted, as none of the 
twenty-five projects between 2003 and 2016 operated on the national level 
(World Bank 2016). Indeed, the Brazilian interviewees described how inter- 
national organisations had redirected their activity towards sub-national actors, 
because the national level had developed capacity and funding independence 
and had different political motives. A new QAE policy expertise started to take 
shape with the help of independent analysis, greater independence in funding, 
and increased political independence, which eventually changed polity. 

Brazil appears to have attained a national capacity for independent data anal- 
ysis in the 2000s, meaning that international organisations’ technical assistance 
was no longer needed. Indeed, Brazilian governments had implemented large- 
scale assessments in the 1990s. Interviewees from different branches of an inter- 
national organisation discussed this shift to working with sub-national instead 
of national actors: 


We haven’t had the dialogue at the federal level that we would like in the 
past five or six years, but we’ve been very active at the state level. 


(BR-IGO-01) 
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Here in Brazil, fifteen years ago, we were doing education projects at the 
federal level ... We don’t do that any longer. They don’t need us. 
(BR-IGO-02) 


Besides the development of national capacity, the international loaning organi- 
sations, the World Bank and the Inter-American Development Bank (BID), 
pointed to funding independence at the federal level as one of the main reasons 
for the change of focus to the sub-national level. Indeed, whereas some inter- 
viewees pointed to the strong national capacity emanating from the National 
Institute for Educational Studies and Research (INEP: the office responsible for 
large-scale assessments), the international organisations did not trust the imple- 
mentation capacity of all the Brazilian states. Unlike the federal government, 
they felt financially strapped states and municipal level governments especially 
needed both financial and technical advice. 


In terms of financing the federal level doesn’t need [name of an interna- 

tional organisation]’s financing. This is different from twenty or twenty- 

five years ago. The state and municipal levels still need financing. 
(BR-IGO-02) 


Basically, one of my first impressions when I came to the [IGO] was that 
obviously you want to help the poorest states. 
(BR-IGO-06) 


The need and potential for political independence at the federal level seems 
also to have been behind the international organisations’ new sub-national 
activity. A Ministry of Education representative pointed out that international 
organisations like the OECD and the World Bank struggled to understand 
Brazil’s complexity, making them less attractive partners. 


I would love to work with anyone who can understand that we are very 
diverse and there is no magic solution for anything ... If the World Bank or 
the OECD could understand this I would be happy to work with them ... 
If the World Bank wants to do this, I’m a partner. 

(BR-N-10) 


A representative of an international organisation noted that ministers were per- 
haps unhappy with some of the results of their reports. 


I’m not sure, but maybe the last ministers did not agree with the evidence 
that the [international organisation] supports. Maybe. 
(BR-IGO-03) 


This political stance is further visible in the different emphases of interna- 
tional organisation and Ministry of Education interviewees concerning whom 
the states should consult in their technical work. The former emphasised 
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international organisations’ technical expertise, whereas the latter saw universi- 
ties as suitable partners. 


I think that sometimes one of the things that [international organisation] is 
doing is helping some of the states, as we have done in Rio, to strengthen 
the statistical and planning units within the secretariats so they can be more 
effective. 

(BR-IGO-06) 


We have to evolve to a model of more partnership so the universities play 
the most important role, and the states and cities of course don’t have many 
technical people to work with us: we have to make an effort to understand 
why they need something, and the university can provide these services 
better than they can. 

(BR-N-10) 


The changes in the Brazilian political arena indicate the gradual accumulation of 
national QAE policy expertise and the way in which this has sidelined interna- 
tional organisations at the national level. Financial capacity and the potential for 
using knowledge are essential in the quest to become a relevant actor in the Bra- 
zilian system. The interviewees describe how the national government’s depend- 
ence on international expertise diminished after it had succeeded in developing 
financial and technical QAE resources. In Brazil, QAE thus appears to be related 
not only to technical and financial but also to political independence. 

In China, the development of expertise is advanced, but there is also some 
caution concerning outside political influence. The events of 1989 proved a 
turning point, after which China became more reserved about foreign educa- 
tion patterns (Altbach 1991). China remains ambivalent about international 
cooperation because it is seen as presenting a potential threat to the party-state’s 
sovereignty (Kent 2002). Although education issues are not especially sensi- 
tive in China, in recent years, the government has become especially cautious 
about INGOs operating in China (Yin 2009; Hsia & White IN 2002). As Yin 
(2009: p. 534) argues, “While recognizing the potential contributions of foreign 
NGOs, the Chinese government is also becoming increasingly wary of the 
growing presence of foreign NGOs in the country and has warned against the 
‘potential national threat’ that they may pose.” Our interviewees also confirmed 
this increasing suspicion, although they did not elaborate why this had been 
the case either with their organisations or the education field in general. Using 
Baumgartner and Jones’s concepts, it is clear Chinese policy seeks to maintain 
apathy regarding assessments. 

Of course, none of this is to say that internationally focused work has had no 
impact on China. First, the country has been influenced by personal experi- 
ences and contacts. The assessment system especially, as has been mentioned, is 
largely staffed by people familiar with foreign practice who have been involved 
in extensive exchanges of ideas. For example, there has been cooperation at 
workshops, lectures, and consultations with people working with the OECD’s 
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Teaching and Learning International Survey (TALIS), the Educational Testing 
Service, and the International Association for the Evaluation of Educational 
Achievement. Foreign scholars also increasingly travel to China as guest speak- 
ers and visiting scholars. 

Second, there have also been more direct contacts. IGOs are directly con- 
sulted, for example, when national education reform plans are drafted to assist 
in their incorporation into China’s international commitments. UN agencies 
seem especially to be in a position in which they are seen — as one inter- 
viewee (CN-IGO-01) put it — as more “governmental” and are thus consulted 
at an even earlier stage of policy formulation than previously. Most recently, the 
OECD, with the World Bank and other IGOs, participated in the preparation 
of the National Guidelines for Medium- and Long-Term Educational Reform 
and Development, which outlines the future development of education for 
2010 until 2020 (Gu 2010). Our interviews confirmed this also applies to the 
next development plan, which extends to 2030 and is under construction. How- 
ever, because the Chinese government has had a quite straightforward vision of 
what counts as quality in education, IGO’ consultative role is primarily to plug 
the gaps beyond the room for state action which the national QAE systems do 
not cover. In implementing their own programmes, IGOs need to align their 
goals to meet the goals of national reform programmes. In addition to IGOs, 
both semi-governmental and non-governmental international development aid 
organisations (we refer to all these actors as INGOs) have been active in China. 
The Chinese state does not allow INGOs to operate independently. They are 
required to work with local state education bureaus. Moreover, except in cases 
where support is limited to the provision of funding, INGOs can only oper- 
ate on sub-national levels, that is, with provincial, county, city, or school-level 
actors, and they align their development goals and plans with national plans. 
The role of INGOs has also been restricted to ensure quality beyond the room 
for state action — much as is the case with IGOs but to an even greater extent — 
and INGOs have thus focused on issues such as sex education and education 
for disabled children. These notions of quality beyond the room of the state are 
also reflected in the data used to ensure quality (CN-IGO-01). 

IGO representatives, however, fully acknowledge that they lack the state’s 
capacity in data gathering: 


I think, of course, we may not do [as] well . .. as [state] research institutes in 
terms of building the evidence. That’s where we have to learn. 
(CN-IGO-04) 


The nature of international experts’ roles and communication with them has 
changed in Russia. No large international organisation currently holds a vis- 
ible position in Russia, and the World Bank no longer contributes to Russian 
education reforms. Much as in China, it could also be said that testing as one 
element of QAE has been more influenced by cooperation with specialised 
agencies (e.g., ETS, CITO, or Cambridge Assessment) and individual experts 
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than by any large-scale, externally funded education reform. However, coop- 
eration still occurs in the context of the transnational QAE agenda outlined 
in Chapter 1. Russian experts and civil servants participate in various inter- 
national activities at their own initiative. Currently, cooperation with interna- 
tional actors seems to happen mostly around the international learning tests 
(PISA, PIRLS, and TIMSS), as the Russian experts who administer and analyse 
these data have become members of executive boards or subject committees 
and/or participate in international conferences and meetings of assessment spe- 
cialists and researchers. However, these tests are not understood as changing the 
system in a top-down manner: 


Interviewer (I): So, the reorientation took place under the influence of par- 
ticipation in international assessment. Or is participation in 
these assessments the result? 

Respondent (R): Reorientation did not happen for this reason; it took place 
because schools in our country started to change ... We 
can’t say the Russian school started to teach children to apply 
knowledge because of international surveys. No, that would 
be a complete lie. The truth is that schools started to change. 
We ourselves understood what was important, that children 
should not learn by heart, and need to apply what they learn 
more. Because our economy is changing, it is becoming inte- 
grated into the world and so on. And only on this basis do we 
take part in surveys which compare us on the basis of criteria 
we understand (RU-E-12). 


In all three contexts, the transnational agenda for the production of data has 
affected the national polity. In Brazil, increasing capacity has resulted in less 
dependency on data and resources provided by international organisations. In 
China, international organisations never gained a strong position in the provi- 
sion of data or resources but were instead confined to indirectly influencing 
decision-makers and focusing more directly on issues deemed marginal in the 
drive for education reform. In Russia, international organisations’ role is less 
than it was in the mid-2000s. The change in QAE agendas has therefore led to 
the development of national QAE expert communities which rebalance state 
actors’ roles (this redefinition of state and expertise is analysed more thoroughly 
in Chapter 5). In this process, national governments have enhanced their data 
analysis capacity to meet the level of international organisations. At the same 
time, data experts have gained more autonomy. Nevertheless, international 
organisations have been increasingly sidelined as national capacity has grown. 


New actors 


In the previous section, we described how QAE actor constellations were rede- 
fined following a change in national agendas. This has affected polity, as it has 
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changed the role of other actors in each of the countries. All have witnessed 
the growing importance of organisations responsible for large-scale assessments, 
which we address in more detail in Chapter 5. There are also varying degrees to 
which the different actors — sub-national governments, international and inter- 
governmental organisations, and national and local non-governmental actors 
(non-profit organisations, commercial companies, professional networks) — find 
room for action with the help of claims for data expertise. 

Our analysis shows that the intertwining of changing agendas and actor con- 
stellations is visible in how QAE expertise and capacity are essential to justify 
who may belong to the expert community. This is true of all our case countries, 
even though they have quite different governance structures. In Brazil, China, 
and Russia, new expert agencies responsible for conducting large-scale assess- 
ments have been established. It is also of interest that we can observe other 
emerging actors such as third-sector movements (Brazil), think tanks (China), 
and fluid expert networks (Russia), reshaping the education-political landscape. 
However, these attempts of unofficial non-governmental or semi-governmental 
(China) organisations to gain greater prominence in decision-making with the 
help of QAE expertise do not always succeed. 


Brazil: ascendancy of civic movements 


NGOs in Brazil are well networked with both public and private actors at the 
national and international levels: they occupy a firm position as part of the QAE 
policy actor constellation. QAE policy and practice have shaped and fed their 
agendas, which are closely related to government QAE policy and practice. 

Social movement organisations, such as Campanha Nacional pelo direito a Edu- 
cação (Campanha: the Brazilian Campaign for the Right to Education) and 
Todos pela Educação (TPE: All for Education), function as major platforms and 
have a decisive impact on policymaking (see Centeno, Kauko, & Candido 2017). 
These NGOs bring together almost every actor, national and international, and 
public and private, and therefore possess significant legitimacy. Furthermore, 
they both exert their influence by lobbying government and producing studies, 
proposals, and activities, as well as by networking and cooperating with major 
national public actors like the INEP and the Ministry of Education and inter- 
national and national organisations. 

QAE has paved the way, either triggering or making possible their involve- 
ment and positioning in the education arena, both in terms of knowledge pro- 
duction and policymaking. As one interviewee explains: 


[T]here are several social actors which had faded a little [from the develop- 
ment of QAE] and [others] which were not organised [but] which emerged 
with that movement [of large-scale assessment], particularly Todos pela Edu- 
cação which was the social framework of legitimisation for the Haddad 
[Minister of Education] project [of the large-scale assessment indicator, 
IDEB.] A Campanha ... which is the bigger [social] movement . . . didn’t 
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have [large-scale] evaluation on its agenda at that time ... [It has] taken [it] 
a long time to construct that vision of the SINAEB, ... the [recently pro- 
posed] National Evaluation System of Basic Education which was a painful 
process [for Campanha] ... with a big effort of [Campanha] ... but Todos 
pela Educação emerged within that framework of [large-scale] evaluation. 
(BR-NNGO-04) 


Privately funded foundations are part of the new but vibrant Brazilian civil 
society. They are important elements of QAE networks. Besides exerting their 
influence through Campanha and TPE, they also have a voice of their own 
and occupy an important place in the political arena. Nationally, they influ- 
ence agenda- setting through knowledge production, vigorous networking, and 
media influence: 


[An] important group that has organised itself recently are the founda- 
tions of major corporate groups ... They are groups based on a vision of 
demanding results. They have organised themselves to say “Look, we have 


to have the children finishing elementary education”... This group has 
grown ...It plays an important role and is very present in the media. 
(BR-N-03) 


At the sub-national level, privately funded foundations influence local govern- 
ment projects through their technical and financial capacity. These founda- 
tions can gather various domestic and foreign sponsors and IGO funding under 
national projects channelled to schools and individuals, the details of which are 
published on their websites. [GOs acknowledge the foundations’ strength and 
engage actively with them: 


And in Brazil most of the research I’m doing is with foundations like the 
Lemann Foundation and the Instituto Unibanco in Sao Paulo. The Ayrton 
Senna Foundation too. I mean, there are some which have really good peo- 
ple who know education and provide a lot of technical help to the states 
and municipalities. 


(BR-IGO-05) 


A need for quality expertise was the catalyst in the establishment of the Bra- 
zilian Association of Educational Evaluation (ABAVE). The scholars involved 
found it difficult to access the main academic forums and were constantly 
struggling to convey the importance and meaning of the quantitative approach 
and its tools. Founded in 2003, the association has gained enormous influence 
on QAE policy and practice: 


[In Brazil] we had the National Association of [Graduate Studies and] 
Research in Education [now the ANPEd]. At some stage that Associa- 
tion split because a group of scholars [who were at the genesis of the 
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large-scale assessments] could not find a way to engage in dialogue within 
an association of education researchers with a broad focus on the Faculties 
of Education ... which are resistant to this framework. So, the spilt was 
a demonstration, and today ABAVE ... is one of the organisations with 
which ... [the INEP] dialogue][s] a lot. 

(BR-N-01) 


In Brazil, the QAE actor constellation has consisted not only of state actors 
but also third-sector organisations and movements. Indeed, the QAE agenda in 
Brazil has stimulated actors’ traditional roles and resulted in the heavy involve- 
ment of the third sector and business interests. 


China: “scholars doing scholars’ work”? 


The creation of the national QAE system in China has given room for action to 
existing actors and given rise to new ones. First, the construction of the assess- 
ment system has transferred power to sub-national education bureaus in rela- 
tion to their corresponding financial bureaus. Under the communist party-state 
system, education, like other politicised spheres, is based on long-term planning, 
and it is the ministerial staff who are tasked with policymaking. High-ranking 
officers in the ministries have had much say in how things are done. However, 
although policies are decided at national and provincial levels, where funding 
and policy implementation are concerned, the regions themselves have con- 
siderably more responsibility (Postiglione 2011; Qi 2011). Many of our inter- 
viewees identified a trend towards such policymaking decentralisation. More 
important, this was also the case where monitoring and assessing quality in 
education were concerned: 


[W]hen it comes to basic education [i.e., the nine-year compulsory educa- 
tion system] quality monitoring in China, we have come up with a say- 
ing: “make counties the system of basic education quality assurance”. This 
means that wherever you go, the county level carries out basic education 
quality management and control. Province and national level governments 
will formulate macro-level supervision policies, but the concrete executive 
units are the counties. 


(CN-E-13) 


As national assessments have started providing feedback to the localities, the 
local educational bureaus, as one interviewee pointed out, have used it as evi- 
dence when they have confronted the financial departments at corresponding 
levels and demanded resources to tackle the problems they have identified. 
Second, and perhaps more important, a notable change in actor relations has 
come with the emergence of actors which are considerably more independ- 
ent of the party-state’s control than previously. In QAE policymaking, these 
developments have manifested themselves in two ways. As has been mentioned, 
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both IGOs and INGOs are consulted in the policymaking process, especially 
when the government is addressing what it considers more marginal issues in 
its reforms plans. In addition to direct consultation, the staff working for these 
organisations suggested when interviewed that it is hoped that decision-makers 
themselves will learn from their initiatives. The logic is that policymakers either 
observe from a distance how projects are faring or that the localities themselves 
or the media pass the information upwards. The INGOs also seek to engage 
directly with staff from domestic NGOs, as well as practitioners and academ- 
ics from the government system whom they perceive as influential. Despite 
this, the organisations’ personnel acknowledge that their influence on policy is 
somewhat limited, except perhaps where changes in practice in the localities in 
which they are directly working with education issues are concerned. 
However, the construction of the national QAE system has made data experts 
more politically relevant. First, provincial institutes of education research have 
been established. These researchers are tasked with researching education issues 
and submit their work to the corresponding executive departments to assist 
in policymaking. Second, the Ministry of Education has established “national 
bases” in universities; these are essentially think tanks tasked with, as one inter- 
viewee (CN-E-05) put it, the “study of policy and study for policy”. The staff 
working at these bases are academics, and many naturally have extensive inter- 
national experience. Because of the traditional emphasis on ranking, much 
attention has been paid to the OECD’s PISA rankings, and the different nations’ 
PISA success propels research interest towards certain education regimes. These 
universities have also established designated branches to research international 
organisations which have strengthened their links with IGOs like the World 
Bank, UNESCO, and the OECD. However, it is noteworthy that the Ministry 
of Education’s bureaucracy does not explicitly bias research foci towards cer- 
tain national systems or research topics. Institutes of education research and 
university-based think tanks produce research data on international organisa- 
tions and assessment practices, which are then utilised to assist in decision-mak- 
ing when policies are formulated. A member (CN-E-07) of such a think tank 
explained, “As a matter of fact our unit is a think tank of the state’s Ministry of 
Education. That means that every time they introduce a policy, we need to do 
research.” Although these expert institutions cannot operate entirely indepen- 
dently of the party-state, their staff consists of academics rather than bureaucrats 
or party functionaries. This emergence of a semi-state approach represents a 
major departure from the days when the Ministry of Education was in abso- 
lute control: “Many people in the Ministry of Education now say that schol- 
ars should do scholars’ work, administrators should do administrators’ work” 
(CN-E-02). However, despite ambition to change the situation, the influence 
of scholars on policy is still somewhat marginal. One interviewee complained, 


[W]hen formulating Chinese national policies and school policies, the 
whole process from formulation to implementation is closed. In recent 
years it has become more open, but despite this, many scholars’ viewpoints 
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hardly ever become policy in practice. The state is vigorously pushing for- 
ward this aspect [of including scholars’ perspectives]. 
(CN-E-13) 


Indeed, the model of policymaking involving prominent individuals within 
the ministry is still dominant, and policymakers rarely have the inclination or 
time to engage deeply with academics and experts. Consequently, communica- 
tion between the groups is often limited to very short policy recommendation 
briefings. 

All in all, and much in line with the technocratic tendencies at play in wider 
Chinese society, normative power has been transferred from party-state func- 
tionaries to academics, whose professional networks and research foci position 
them firmly within a transnational network. As our analytical framework sug- 
gests, this reveals the changes QAE has made to the actor constellation: where 
the question of who participates in defining and assessing quality is concerned, 
emerging actors are challenging the existing status quo. An interviewee draws 
an interesting parallel with Chinese education reforms, in which relevance also 
equates to financial gain: 


These things [reforms of the QAE system] benefit certain people. Actu- 
ally, it is not just education, but in other aspects the deep reforms in China 
touch certain interests. But these reforms have to be made and there is no 
other option. If not, there is no development ... If we take the reform of 
China’s state-owned enterprises as an example, they were all monopolies 
before. Communications and petroleum [industries], for example, were 
all monopolies. And monopolies do not have competition, and without 
competition the results will always be very poor. If the state’s inputs were 
very big, it was possible that the profits were also very high, but all these 
profits were made by monopolies. So, these state-owned enterprises were 
reformed by dragging them into a system of competition. But this of 
course did not go well with some people, because previously they were 
able to make a lot of money, and this changed when private enterprises 
came into the picture. Other people took their slice of the cake. This is also 
the case with education. 

(CN-S-01) 


Such an approach indicates, implicitly at least, that relevance is not — and should 
not be — only a top-down decision, but different actors’ perception of quality 
(and how it should be assessed) should compete for relevance in the QAE field. 


Russia: state-dependent QAE 


Compared with Brazil, Russian QAE reforms and agenda-setting are more 
state-driven and in this sense closer to the Chinese constellation. Where the 
agenda is concerned, the recent change in Russia has been significant. Previously, 
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Russia only had one standardised test at the end of grade 11 (the Unified State 
Exam, USE), which became compulsory throughout the country in 2009. The 
USE is both a school graduation and university entrance examination. Along- 
side more general education statistics, the scores of the USE have constituted 
a key source of information for national system evaluation and evidence-based 
policymaking. The Russian federal authorities are now seeking to reinstate a 
“uniform educational space” for education. This entails the introduction of 
new federal curricular guidelines which define the expected learning results 
and development of more diverse QAE procedures initiated by the national 
authorities. 

The federal level has introduced another standardised graduation examina- 
tion after grade 9 (at the end of lower secondary, or basic, education). A sample- 
based National Study of Education Quality (NIKO) was also initiated in 2014 
with a focus on primary and secondary school subjects and their respective 
skills. It was introduced with the intention of grasping how children are taught 
in different parts of the country to identify problematic areas and best practices 
and to compile nationwide education statistics. The federal authorities particu- 
larly emphasise the difference between NIKO and the high-stakes USE, main- 
taining that the former poses no risk to the students and schools assessed in the 
survey. Since 2015, the federal level has also started to administer the All-Russia 
Examinations, now being piloted and planned as part of the overall evaluation 
system in the near future. They test end-of-school-year learning results annu- 
ally, and these are said to provide information on learning achievements for 
school, local, and federal actors. 

These different data sources form the basis of a large annual report about the 
state of the education system, prepared by a department of the federal ministry. 
The first such report was published in 2015 and based on information collected 
in 2014: 


Well, this is a bit of an analogy to what in some countries is called a 
national report on the system of education. We had nothing like that [for 
about 10 years]. We only collected statistics in one way or another, and 
they were utilised locally somewhere for developing some local projects. 
But from last year [2015] it became a systematic process, and we write a 
serious report about all levels of education on the basis of statistical data, 
data we receive from the federal units on the basis of a particular plan of 
the characteristics of the system. It is not only a statistical database, but also 
an analytical database that detects some trends, some problematic areas, and 
some serious changes affected by the decisions taken earlier. 

(RU-N-03) 


It is clear from our interview materials that it is planned to make the collection 
of data about the education system more detailed and systematic, which will 
have further effects on the actor constellation. When our interviews were con- 
ducted, the federal ministry was in the process of establishing a blueprint for how 
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the different sources of information would feed into national decision-making. 
Indeed, the federal government is the main architect in designing changes to 
the QAE organisations. The recently established Federal Institute of Education 
Quality Evaluation is intended to become a centre for data accumulation and 
processing under the auspices of the Federal Service for Supervision in Educa- 
tion and Science, which means the Federal Ministry of Education and Science 
is its superior. Moreover, the federal level is dissatisfied with the quality of 
information it receives and is seeking to establish a system that will help obtain 
more detailed knowledge of schools: 


[T]he subjects of the Russian Federation are in charge of collecting school- 
level data, so that it is aggregated on the level of the federal subjects and 
presented this way to the ministry. So, this means that we don't see the con- 
struction inside the school. This is the first problem and we are aware of it, 
so from January we will introduce a system that will enable us to establish 
an infrastructure for, from next autumn and winter, gaining information 
about each school. 

(RU-N-03) 


The developments just described signal a strong desire for a nationally 
anchored centralised and standardised education evaluation system. This entails 
increasingly centralised data collection and analysis by actors accountable to 
and financed by the national authorities. At the same time, expert networks 
also play an important role in Russia. The predecessors of the USE and the 
structures administering them were small-scale commercial organisations and 
research groups interested in standardised testing. They learned the craft of 
standardised testing from available international sources because of their per- 
sonal or commercial interest, aided by their prior training in mathematics and 
psychology, for example. The historical roots of the two ministerial units now 
running the USE for grades 11 and 9 are in the early post-Soviet practice of 
voluntary standardised testing of university applicants. These skills and wider 
experience were utilised when standardised assessments became a federal initia- 
tive, promoting experts with the required skills to the higher ranks of the newly 
established federal structures. 

An equally important and partly connected group of evaluation experts 
began developing their evaluation proficiency in direct connection with the 
international tests in which Russia had first participated — TIMSS in 1995 and 
PISA and PIRLS in the early 2000s. These experts later performed important 
roles as test developers and analysts in the USE. However, as the following 
quotations demonstrate, they did not take the transfer of foreign expertise for 
granted, and they quickly developed their own capacity: 


Then we started, . . . this is the very first point [to consider], ... what can 
and cannot be transferred. We learned ... how to do it — we even invited 
specialists from the Netherlands who developed PISA 2003. They held a 
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wonderful seminar. Almost all the subject specialists who write test items 
for the USE took part in the seminar. It was great, interesting, but we could 
not utilise the test tasks themselves. 

(RU-E-12) 


We knew the state of affairs in Russia. Unfortunately, there were situ- 
ations when our colleagues, especially Americans, thought that we had 
just jumped off a branch of a tree . .. Nevertheless, of course, we learned 
something, and what we told them about what we have done was interest- 
ing for them too. 

(RU-E-10) 


Overall, links between national and international experts are highly appreci- 
ated, which also means that policy transfer has occurred through professional 
interactions the Russian actors for the most part perceive as normal mutual 
learning and professional communication. Testing expertise has been the most 
significant lesson: 


You understand, the instrument is very new for this sphere. And we would 
not have created it had the Bank not attracted an excellent consultant ... 
Without [this person] we would not have created the instrument: [he/she] 
is the ideologue behind everything, and through [him/her] all this knowl- 
edge, skills and competences in creating test items, processing of results, 
the approaches and methods we use in the developing of the tasks and the 
assessment scale, all of this is due to the correct scientific advice of [the 
consultant], no doubt. 

(RU-E-07) 


Many if not most of these Russian experts have stayed with the organisations 
running standardised assessments and quality evaluation by circulating within 
and between different organisations and blurring their identities as civil servants 
or academics at different phases of their careers. Overall, while in Russia the rise 
of QAE has made expertise and the individuals possessing it more significant, 
the demand for education quality assessment and data production has not led 
to the emergence of structures independent of the national authorities. Instead, 
national policies and goals permeate all the new establishments, even if their 
functioning was initially enabled by and relied on the expertise of qualified per- 
sons located outside the ministerial apparatus and connected with transnational 
professional networks. 


Conclusion: a transnational expert community 


In the preceding examples, the most dramatic changes in actor constellations 
after the introduction of QAE agendas happened in Brazil, where we observe 
a fundamental change of thinking concerning which actors are relevant in 
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education policy. In the Chinese and Russian cases, the state authority has 
maintained a tighter grip on what is happening in the field of QAE in edu- 
cation. Baumgartner and Jones (2009) suggest the main vehicle for change is 
positive, change-supporting feedback. In Brazil, the state does not exert tight 
control over QAE policy, as there are many third-sector movements contribut- 
ing to the agenda. The feedback provided by Brazilian civic movements would 
not be possible in the state-controlled systems of China and Russia. In China, 
for example, the results of large-scale evaluations are not disclosed to the public, 
which does not encourage the demand for improvement in schools’ quality. 
However, we do observe a change of agenda, effected by international organisa- 
tions and the state-led creation of the QAE expert community. 

This chapter also indicates the way in which the case countries were able to 
develop more QAE capacity during the 1990s and early 2000s. This enhanced 
data infrastructure and expertise have made national governments less depend- 
ent on international organisations, but they have also remained interested in 
international trends. Despite the bigger picture of the state retaining control 
of QAE in China and Russia, all our cases have seen change. Brazilian sub- 
national actors, as opposed to national actors, now work with international 
organisations and their loans. The QAE infrastructure has also allowed civic 
movements to play a more active role in education. In China, some semi- 
governmental organisations, such as think tanks, have become more significant 
as QAE has gained in importance. In Russia, room for action is largely reserved 
for national agencies and their various tests. However, expert networks play 
a role in channelling external influence. The role of international actors has 
changed in all contexts, and in all cases, the countries themselves have been able 
to direct their influence. 

In the analysed cases, there are clear differences in how the transnational 
QAE agenda has affected national polity. However, a common dynamic is 
involved in how transnationalism becomes embedded in expert communi- 
ties. Although the attitude towards international advice has historically varied 
between and within each country, expert networks are increasing the transna- 
tional information flows. This change has occurred gradually, but it is visible 
in each context. This transnational embeddedness is also supported by the 
fact that international organisations have sought to intensify their sub-national 
operations in Brazil. In Russia, the international organisations’ operations were 
largely sub-national during the early post-Soviet period, but their overall role 
has since been significantly reduced, including at the sub-national level. In 
each case, the change of agenda has been connected to a change in actor 
constellations. 
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Introduction 


Although the state is still a relevant and powerful concept in legitimising actors 
and directing policy, legitimation through data ownership and experts’ instru- 
mental knowledge has become a means of finding more room for action. Sev- 
eral studies have affirmed the role of knowledge in the shaping of education 
policy and implementation of education reforms on both national and global 
levels (Normand 2017; Lawn & Normand 2014). Expertise plays a crucial role 
in the production and interpreting of knowledge. It is “a specific activity of 
knowledge production participating in the process of negotiation and ori- 
entation of public policy” (Normand 2017: p. 74; see also Ozga 2011). This 
knowledge is mainly technical and prescriptive/normative and derives from 
the “professionals working in administrations of the states, international organi- 
sations, universities and the HE institutions, agencies, think tanks or interest 
groups” (Normand 2017: p. 74; see also Weible 2008). Policy arenas like con- 
sensus conferences are also central to the academic production of knowledge 
with various structures and conditions of reception (see Weiss 1979). 

In some epistemic communities, experts are mandated and certified to par- 
ticipate in the entire collective learning process. Policymakers thus call on 
expertise in seeking to reduce uncertainty about information and knowledge 
in decision-making processes. As Normand (2017: p. 75) writes: “Experts are 
controlled because it is policy-makers who make the decisions regarding the 
modalities of dissemination and the use of knowledge. Experts have to respect 
precise specifications defining the modalities of reciprocal learning and knowl- 
edge production.” Such policy-expert acquaintances are a means of producing 
knowledge based on personal relationships between experts and policymakers 
which differs from the division of labour orchestrated by knowledge centres for 
the production of evidence-based resources. Types and organisations of exper- 
tise differ significantly in serving policymaking. 

In some cases, experts have considerable capacity to anticipate emerging 
political issues because of their previous political and administrative experi- 
ence and capacity to represent and lead important expert networks. The capac- 
ity of anticipation has also shifted from states’ and international organisations’ 
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expertise in traditional planning and prospecting to international consultancy 
groups like McKinsey or Pearson, with the capacity to elaborate short-term 
scenarios and knowledge-based technologies and resources. Depending on 
their influence on and closeness to the media, experts affect the communi- 
cation and interpretation of selected research findings and studies differently 
from various groups and audiences. They usually strive to directly influence 
the media and specific target groups “by adopting a clear, concise and under- 
standable language while using media devices at their disposal (blogs, Facebook, 
Twitter, etc.)” (Normand 2017: p. 76). 

This chapter analyses and contextualises the changing role and relationship 
of the state and expertise in Brazil, China, and Russia, when large-scale policies 
of quality assurance and evaluation in school education have been undertaken. 
Our main question concerns the nature of expertise in reshaping the state's 
context, status, possibilities, and limits in governing education reforms through 
quality assurance and evaluation (QAE). In contrast with Chapter 4, this chap- 
ter focuses more on the national level than international organisations. Chap- 
ter 6 returns to a closer examination of the processes experts conduct. 

Usually, the state's interest and approval seek to determine the development 
and application of QAE, and this is also empowered and supported by QAE 
data. However, the data generated by various instruments and databases greatly 
depend on experts, who in many cases generate the lexicon and concepts of 
data use. Independent experts devise their own instruments and data analysis, 
but state authorities and structures create a wider, but also restrictive, demand 
for data collection and use. Concurring data collections exist, which seek to 
adjust the information collected but which may overload the system. Experts 
may therefore find their room for action diminished. There is a general mis- 
match between the demands of experts and state officials. Independent experts” 
access to state-owned data is also often limited. The roles of and relationship 
between the state and expertise seem to vary greatly in Brazil, China, and Rus- 
sia. To tackle these differences, we ask the following sub-questions: 


e Who are the main experts in Brazil, China, and Russia recognised as 
responsible and legitimate actors to collect and interpret data in the QAE 
of school education policies? 

e What kind of relationship and division of labour is there in Brazil, China, 
and Russia between the state and experts, and is there evidence that this 
relationship is changing? 

e Are the experts and/or state authorities and policymakers willing to limit 
and restrict or widen and open data use in their practice of the education 
policies grounded in QAE? 


In the next section, we analyse the state’s changing role in the historical and 
social situation of the twenty-first century, which has seen growing global, inter- 
national, and transnational influences and impacts on nation-states. The third 
section discusses and analyses experts’ growing and changing roles. Our main 
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theoretical and empirical points of departure here derive from the wide and deep 
research tradition of sociology, the political sciences, and comparative educational 
science in the fields of the state and the changing role of expertise. Most of the 
research we use here has been undertaken by Western researchers within the 
frame of the Western research tradition, and it is also interwoven with the views 
of international organisations such as the Organisation for Economic Coop- 
eration and Development (OECD) and the European Union and the Western 
worldview. In this respect, they are manifested in many ways in the relatively rich 
welfare states and may contain angles and perspectives which inadequately fit 
the circumstances, economies, and social, cultural, and educational heritages of 
the “emerging countries”, to which Brazil, China, and Russia are often said to 
belong. We are deeply aware of this potential bias and seek to address it seriously 
in our framework’s critical challenge, theory formation, discussion, and analysis. 
After the theoretical sections on state and expertise, the following three sections 
analyse and seek to answer our research questions empirically, drawing largely on 
interview data from each of our case countries. The final section summarises our 
comparison of Brazil, China, and Russia and takes issue with the more theoreti- 
cal discussion about the changing relationship between expertise and the state, 
using the idea of CADEP to understand actors’ roles and dynamics. 

We conclude by reflecting on the dynamics of attempts to govern the edu- 
cation policies of nation-states with evidence-based technologies through 
data, numbers, and indicators. The state’s role is also shaken by the many pro- 
found changes within the countries, including the partial loosening of its strict 
monopoly of collective decision-making and the new pressures to decentral- 
ise and deregulate it, which creates more room for different actors and actor 
groups, as well as the growing number of experts. 


The changing role of the state 


Bauman (1987, 1992, 1998, 2004) has replaced his older concept of “post- 
modernity” with the concept of “liquid modernity”. He emphasises that there 
are no stable institutions or conditions in the global age and that there are no 
frames because everything everywhere is constantly in process. “While trust 
and confidence were constitutive of early modernity, risk and uncertainty are 
now the hallmarks of liquid modernity” (cited in Kwiek 2006: p. 306). In liquid 
modernity, the traditional post-war Keynesian welfare state, with its powerful 
state institutions, is questioned. Current transformations are giving birth to 
profitability and to “a new kind of capitalism, a new kind of economy, a new 
kind of global order, a new kind of society and a new kind of personal life” 
(Beck 1999: p. 2, cited in Kwiek 2006: p. 27; Rinne & Ozga 2011). Where our 
chapter’s focus is concerned, this should also affect the relationship between the 
state and experts. 

These changing questions of power boil down to the term “governance”, 
which is used in many social science disciplines as an interdisciplinary bridg- 
ing concept (see Schuppert 2006: p. 373). There is no agreed-upon definition 
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of the term: governance is used to describe new forms of steering/regulation 
(Benz 2004). From the outset, it is important to note that governance is not 
a narrow theory but rather an analytical approach which draws attention to 
important changes in political perspective. These shifts in perspective concern 
the use of governance to conceptualise regulatory and governance activities 
which concepts such as “steering”, “governing”, “control”, and “interdepend- 
ence” have conventionally favoured. Governance addresses “government”, 
“management”, “coordination”, “regulation”, and so on among the different 
actors within the state, market, economy, and civil society non-hierarchical and 
network structures (Benz et al. 2007; Ball & Junemann 2012; Loncle, Parreira 
do Amaral, & Dale 2015). 

Governance emerged as a political concept at a time of criticism and scep- 
ticism concerning the possibilities for a linear and hierarchical regulation of 
complex social systems. This points as much to a change of perspective con- 
cerning forms of regulation and coordination in modern societies as it does to 
any profound transformation (Mayntz 2006; Dale et al. 2016). 

Patrick Le Galês defines the substance of governance as 


[A] coordination process of actors, social groups and institutions that aims 
at reaching collectively defined and discussed objectives. Governance then 
concerns the whole range of institutions, networks, directives, regulations, 
norms, political and social uses as well as public and private actors which 
contribute to the stability of a society and a political regime, to its orienta- 
tion, to its capacity to lead, to deliver services and to assume its legitimacy. 

(Le Galés 2004: p. 243; cited in Dale et al. 2016) 


Changes in governance resulting from the new steering tools the expert com- 
munity usually uses have been widely noted. In Chapter 6, we present a critique 
of Rose’s and Miller’s idea of “steering at a distance”, which follows from the 
analysis of our case countries. However, it is helpful to understand that the 
principles of calculability and measurability originating from economics used 
by the private sector have increasingly been transferred to fields previously 
regulated by old bureaucratic statutes and professional norms, usually located in 
the public sector. Rose (1999: p. 152) refers to the new governance technology 
based on accountability and assessment to which the public sector is subjected 
as “governance at a distance” (Rinne & Ozga 2011: p. 67). This steering has 
consequences, as Rose puts it, for the shift to an “audit society”, where every 
new space subjected to comparability, measurability, and transparency summons 
its population to evaluate and measure itself and translates its activities into 
measurable and economic language to maximise efficiency and income, while 
arbitrary rules are “tamed, liberalized and acknowledged as neutral and objec- 
tive calculation and evaluation” (Rose 1999: pp. 152—154; Rose & Miller 1992, 
cited in Rinne 2001: p. 107). 

Similarly, Michael Power has developed the concept of the “global inspec- 
torate or audit society” (1999, 2003), and he observes that evaluation entails 
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a “control of controls” and “rituals of verification” (cited in Rinne & Ozga 
2011). In political science, the focus on understanding governance structures and 
processes as opposed to governments’ structures and processes occurred more 
than a decade ago. Briefly, this focuses on the shift from centralised and vertical 
hierarchical forms of regulation to decentralised, horizontal, networked forms 
(e.g., Rosenau 1999). Although it is widely acknowledged that governance is 
a phenomenon which produces broad patterns reflecting dominant political 
forces, it is also important to note that it is understood as a continuum which 
“stretches between the transnational and the subnational, the macro and the 
micro, the informal and the institutionalized, the state-centric and the multi- 
centric, the cooperative and the conflictual” (Rosenau 1999 cited in Ozga et al. 
2011: pp. 89-90). 

Where steering tools are concerned, the OECD’s knowledge-based regu- 
latory tools vigorously attempt to promote orthodox professional practice 
and the increased standardisation of professional formation and development. 
The strength and power of these tools lie in their apparently objective nature, 
in the attractiveness of the space for negotiation and debate they create in 
which experts, policymakers, and other knowledge-brokers meet and posi- 
tion themselves, and in their capacity to define the terms of that engagement 
(Rinne & Ozga 2013: p. 97). Pons and Van Zanten (2007) suggest these 
tools have three main elements: (i) they reflect particular “world visions” 
which represent the agenda-setting capacities of particular interests; (11) they 
represent a particular and politically oriented set of beliefs concerning legiti- 
mate policy in a given domain; and (iii) they represent a wide and growing 
network of actors who are constantly drawn into the process of intelligence- 
gathering, auditing, and reflective policymaking (cited in Rinne & Ozga 
2013: p. 97). 

To grasp the implications of the increasing complexity of the emerging 
multi-scalar/multi-level governance arrangements in each country, we need 
to devise a new lens through which to examine the issues. Dale sees this as a 
major shift: 


With new forms of complex governance, the state form ... loses its monop- 
oly position in the production of collective solutions to the collective 
problems. Collectively binding decisions are no longer taken by the state 
alone, or among sovereign states, but rather with the involvement of vari- 
ous types of societal actors, sometimes even without governments. 

(Dale 2009: p. 30) 


Dale and Robertson (2012: p. 23) make a similar argument and emphasise a 
change in the 


national education system to a more fragmented, multi-scalar and multi- 
sectoral distribution of activity that now involves new players, new ways of 
thinking about knowledge production and distribution, and new challenges 
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in terms of ensuring the distribution of opportunities for access and social 
mobility. 
(see also Dale 2003) 


Verger, Lubienski, and Steiner-Khamsi (2016: p. 4) also analyse the growth of 
the “global education industry” and observe that its emergence has entailed the 
development of new market niches “that are often outside of traditional state 
control, such as preparation, edu-marketing, the provision of curriculum pack- 
ages or school improvement services”. 

However, it is important not to exaggerate the state’s defeat here and suc- 
cumb to what Weiss (1997) calls “the myth of the powerless state” (cited in 
Ball 2007: p. 36). Indeed, despite an increasing international interdependence 
which seems to generate pressure to converge, advanced industrial societies 
continue to exhibit differences in their institutional practice. As Andy Green 
(1999: p. 56) observes, 


As regards education, there is very little evidence across the globe that 
nation states are losing control over their education systems or ceasing to 
press them into service for national economic and social ends, whatever 
the recent accretions of internationalism. In fact the opposite may be true. 
As governments lose control over various levers on their national econo- 
mies and cede absolute sovereignty in foreign affairs and defence, they 
frequently turn to education and training as two areas where they do still 
maintain control. 


The orthodox conception of national education systems effectively draws 
together distinct functions and sets of rules and beliefs. Those which have been 


historically rooted in an assumption of the centrality of national economies — 
for instance, that the function of education systems was to provide edu- 
cated labour for the national economy and that education systems would 
shift and separate the potential workforce, according to ability and poten- 
tial contribution to the economy. Another assumption was that education 
systems are crucial to the construction of national culture, integration and 
sense of national belonging. 

(Parreira do Amaral & Rinne 2015: pp. 80-81) 


Our questions concerning the changing role of state and experts in this 
chapter are much the same as Ozga’s (2015: p. 30) note on the paradox of “fun- 
damental commitment to reducing the role of the state and enabling system 
and self-regulation through the market” and “the need to use state regulation 
in order to get the market to function properly”, which in turn “is creating 
constant pressure for increased regulation and centralization”. Citing Kandel 
(1938: p. 29), she warns: “The danger that confronts mankind to-day comes not 
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from the expansion of education, but the specialization in some narrow corner 
of the field of knowledge. The specialist faces the world to-day as the blind man 
the elephant and fails to see life steadily and see it whole” (Ozga 2015: p. 31). 


The context of expertise 


The complexity of governance, and thus the operating environment for experts, 
is increasing. Bob Jessop (2002: p. 199) calls this “destatization”, which involves 
“re - drawing the public - private divide, reallocating tasks, and rearticulating 
the relationships between organizations and tasks across this divide”. Alongside 
these changes global education policy communities are constituted through 
which new policy discourses and narratives flow. The new global policy net- 
works are built by a 


diverse set of think tanks, consultants, multi - lateral agencies, donors, edu- 
cation businesses, and philanthropies, [which] constitute policy communi- 
ties that are based upon shared conceptions of social problems and their 
solutions. New narratives about what counts as a “good” policy are articu- 
lated and validated. 

(Juneman, Ball, & Santori 2016: pp. 537—538; Ball 2007) 


Private providers are now also involved at different levels and scales, through 
advice, consultation, evaluation, philanthropy, partnerships, representation, pro- 
gramme delivery, and outsourcing in the provision, monitoring, and evaluation 
of public-sector services. This has “brought new players, voices, values, and 
discourses into policy conversations” and “governance by networks”: “webs of 
stable and ongoing relationships which mobilize dispersed resources towards 
the solution of policy problems” (Pal 1997; cited in Juneman, Ball, & Santori 
2016: p. 537). The situation is of course somewhat different in Brazil, China, 
and Russia, but most of these private providers have gained a foothold in them 
as well (see Chapters 6 and 7). 

Data and data systems support the growing activity of expert networks, 
which aim to identify and frame policy problems and solutions nationally. Net- 
works of new actors and experts also exist. A major expansion of education 
consultancy and provision of educational goods and services contributing to 
the promotion of standardised testing and “standardizing doxa of best practices” 
is taking place. A range of sophisticated standardisation instruments, quality 
benchmarking, and data harmonisation “underpin[s] the governance turnls] 
and act[s] on and within the national systems promoting ways of controlling 
and shaping national, institutional and individual behavior”. New data enable 
the state to work on schools and localities as a resource for steering state pol- 
icy in its entirety and compel experts to move “beyond the traditional task of 
informing policy and [become] policy forming in a more complex form of 
governing” (Fenwick, Mangez, & Ozga 2014: p. 5). 
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In defining the function of experts, Lawn and Segerholm (2011: p. 45) 
emphasise the transnational nature of their work: 


Experts aid the flow of data across countries, and so act to constitute a new 
spatial infrastructure to emerge around education. Through shared data- 
bases, policy officers, technical experts and meetings, new synapses operate 
through which messages of comparison and commensurability (messages 
of standards, ranking scales, indicators and benchmarks) are passed. 


All this is happening between national agencies and the OECD, which are 
the major players in constructing a common education space through the flow 
of data. Data may be called “the currency of governance”. Similarly, Fenwick, 
Mangez, and Ozga (2014: p. 4) claim that to understand and capture global- 
local interactions, we should take seriously Sassen’s (2010: p. 10) injunction to 
think of the “global — whether an institution, a process, a discursive practice, 
an imaginary — as both transcending the exclusive framing of nation states and 
partly emerging and operating within that framing”. 

In developing countries, education policy has relied on the discursive con- 
struction of state failure (for example, in limited access and poor quality). 
Robertson and Verger (2012) suggest a strong generalisation regarding the 
“developed” world has been accompanied by “a purposeful framing of causes 
and issues (failing state, lazy teachers, lack of incentives, lack of accountability, 
dysfunctional schools) and a selective use of evidence”. These have been articu- 
lated by “neo-liberal rationalities that link market mechanisms such as choice 
and fee payment to greater accountability and education quality” (Juneman, 
Ball, & Santori 2016: p. 538). However, this is not especially characteristic of 
Brazil, China, and Russia, where choice and fee payment are concerned. 

The governance of education is thus increasingly understood as taking place 
through cross-sectoral networks of public, private, and third-sector interdepend- 
encies which crisscross national and transnational boundaries. This “networked 
governance” is decentralised and characterised by fluidity, looseness, complexity, 
and instability (Ozga, Dahler-Larsen & Segerholm 2011; Williamson 2012). 
Post-bureaucratic networked governance is conceived as “soft power”, which 
works 


through techniques of attraction, seduction, persuasion and the cultivation 
of support and shared interest across networks of loosely associated actors. 
Soft forms of governance include self-regulation, self-evaluation, self- 
governance and governing through the capacities of the governed, rather 
than the hard government of centralized targets and external regulation. 
(Williamson 2014: p. 218) 


Our analysis sees experts as actors practising “a specific activity of knowl- 
edge production participating in the process of negotiation and orientation 
of public educational policy” (Normand 2017: p. 74; see also Ozga 2011; 
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Lawn & Normand 2014). In the following empirical section, experts are 
those engaged as consultants, researchers, scholars, and lecturers in the field 
of education. 


The state and expertise in Brazil 


The Brazilian expert community is strong and has already been transnationally 
linked for many decades (Kauko et al. 2016). In Chapter 4, we observed that the 
changes in agenda partly relayed by these networks have transformed the politi- 
cal constellation of actors in Brazilian polity. In this chapter, we focus more on 
the potentially most influential expert body. 

The experts in Brazil recognised as responsible and legitimate actors for 
the collection and interpreting of QAE data in the field of school education 
policy are located in the main national expert body for large-scale assessments, 
the National Institute for Educational Studies and Research (INEP), which is 
responsible for Brazil’s main education indicators, such as the Index for Devel- 
opment in Basic Education (IDEB). There are other important expert bodies, 
third-sector movements, and think tanks in the Brazilian expert community, 
but none surpasses the INEP in its scale of data production, which is why we 
concentrate on this organisation in analysing the changing role of expertise. To 
tease out the answers to this chapter’s central questions, we need to investigate 
interviewees’ descriptions of the scope and potential of the INEP on different 
levels of action. 

Our chapter focuses on the nature of the relationship and division of labour 
in Brazil between the state and experts. We have already adduced evidence 
of this change in Chapter 4. In this chapter, we observe a strong willingness 
to cooperate with the INEP on different levels. Policymaker interviewees 
described the INEP as a close collaborator. The Chamber of Deputies Stand- 
ing Committee on Education, for example, holds expert hearings with the 
INEP and Ministry of Education (BR-N-11, written communication). The 
INEP also closely collaborates with the Ministry of Education in producing 
analytical tools (BR-N-02). However, at the sub-national state level, there is 
more demand for training than the INEP can offer (BR-N-09). In this sense, in 
relation to the scope of the INEP’s operations, we can see that it receives strong 
legitimisation from different actors. 

Our interviewees also pointed to the principle that the Ministry of Educa- 
tion is not allowed to deal with political questions at sub-national levels. One 
interviewee stated that large-scale assessments still provided the Ministry of 
Education with information regarding work in the great majority of munici- 
palities and states but allowed such separation and non-involvement. Indeed, 
the interviewees’ wording indicates there is no willingness in the Ministry of 
Education to use large-scale assessment as a tool: 


The Ministry of Education does not aspire to have more political power 
in the field of educational policies in the area of basic education ... [T]he 
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formulation of large-scale assessments is certainly not a strategy to gather 
more power. 


(BR-E-02) 


In analysing whether Brazilian experts, state authorities, and policymakers are 
willing to limit and restrict or widen and open data use in practising education 
policies grounded in QAE, our interviews suggest that the INEP has more 
power than it currently exerts. However, there were contradictory understand- 
ings of how independent the INEP was of the political process. An INEP expert 
argued for more autonomy for the organisation. This interviewee described an 
incident in which it was alleged that the INEP was postponing publication 
of a report because of government pressure. Our research cannot verify if this 
happened, but it indicates actors’ perception that the Ministry of Education 
retains some influence on the publication of results and that the INEP is not 
completely independent. 


Then it happens, eventually ... The ministry interferes, saying “Wait a 
moment, calm down, for me this isn’t a good time for you to say that, there 
is an election now, in a little while, then no, no, it is not so interesting to 
publish that.’ No one will interfere in an autonomous institution, but it 
needs to be 100% autonomous. With the dissemination of some results, 
last year it was questioned, regarding the deadline, if there was interference 
or not. I would not be able to tell you, but PH tell you that if there's a cat's 
tail, a cat's ear and it meows it is probably a cat. I do not know if there was, 
but there's some reason to think so. I would expect the INEP to be more 
autonomous than that. 

(BR-E-06) 


In analysing experts’ and policymakers’ willingness to limit and restrict or widen 
and open data use in practising the education policies grounded in QAE, we 
found that, although there is no consistent view, the self-understanding of the 
INEP was more that of an expert than a policymaker, which created difficulties 
with political issues, as the preceding quotation indicates. However, some also 
emphasised that the INEP was almost a prerequisite for decision-making and 
that without it there could be no policy monitoring. 


The INEP’s status is as a national agency, meaning it has relative auton- 
omy with respect to the Ministry, but maybe it is one of the agencies 
that suffers a more direct impact from policy decisions. [The INEP is] 
subject to political decisions with little dialogue. It is also my perception 
that there is little dialogue and it feels ... that we are often performers 
and that goes against the identity of the researcher. This is the dilemma 
that we suffer: how can I be a researcher if I feel I am a political per- 
former? It's like I don’t have ...I can think but I can’t perform what 
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I think but what someone has told me to do. It's a pretty important 
ambiguity. 
(BR-N-09) 


The main data of Brazilian education today are produced here within the 
INEP Information is fundamental not only for the elaboration of policies 
but for monitoring of policies. Do you understand, then for example, there 
is work done by the Presidency of the Republic on problems, inequality 
in Brazilian schooling, etc.? They constitute an observatory for the Presi- 
dency of the Republic and the data are collected by the INEP, you under- 
stand, so without the information provided by the INEP you cannot only 
elaborate the policies but monitor them. 

(BR-N-04) 


A different view was offered by a state-level politician, who pointed out that 
a major reform of the matriculation examination (vestibular) was being imple- 
mented without consulting experts. 


In 2009 the Minister decided that he would replace, roughly speaking, the 
Brazilian vestibular but had no dialogue with the team, no internal dialogue 
to find out what this team of researchers thinks about it. 

(BR-N-09) 


It might be concluded that the large-scale assessment data the INEP collects 
allows it potential room for action, but this potential is not realised because of 
the actor relations and set practices just described. However, the organisation 
can address problems through public discussion. Although it was alleged politi- 
cians were trying to influence the INEP, we observed no consistent pattern, 
and in general, the organisation seemed quite autonomous. Chapter 6 points 
out that interviewees reported policy processes in which although the INEP’s 
expertise is available, policymakers do not always consult it in preparing impor- 
tant decisions. Its role is ambiguous: the Ministry of Education’s distance and 
the INEP’s orientation do not support political interference, but in some cases, 
the political process finds its way around the experts. 


The state and expertise in China 


In China, expertise and the state have an entwined relationship that is ingrained 
in the Chinese culture of meritocracy, which “originated in the philosophies of 
Confucianism and Daoism during the fifth and sixth centuries BC and devel- 
oped with the Legalists in the subsequent centuries” (Liu 2016; Yao 2000). The 
Confucian stance on this issue is that an excellent student or scholar should 
pursue a political career and an excellent politician should also become a good 
scholar (see Legge’s translation 1983). Pursuing a political career was the only 
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proper goal of education in the context of Confucianism. Liu (2016) writes 
that the direct manifestation on the institutional level was the civil service 
examination system known as kējú, through which the government selected 
government officials. Those who passed kējń became government officials and 
were given power and authority to govern because of the trust the educated 
enjoyed as a result of their knowledge and expertise, which enabled them 
to better understand how to govern or manage social issues. However, those 
who did not fit into the system often chose to serve the aristocracy as advisers 
mou shi. The civil service examination system was abolished in 1904, a few 
years before the end of the last Chinese imperial era. But the legacy of meri- 
tocracy remains deeply rooted in Chinese culture, and this is manifested in 
the current political system in elite governance or authoritarianism (Han & 
Ye 2017). 

The relationship between expertise and the state best explains the contem- 
porary meritocracy. Our findings reveal that policymaking expertise is largely 
provided by researchers from various domestic and foreign research institu- 
tions and universities. They still play the role of advisers to policymakers, assist- 
ing them to understand what the real problems and possible solutions are. 
The final decision in policymaking remains in policymakers’ hands, however 
(Han & Ye 2017). 

When we seek to identify the main experts in China recognised as responsi- 
ble and legitimate actors in collecting and interpreting data in the field of QAE 
policies, we observe that the state usually collaborates most directly and closely 
with top-ranked universities and research institutes associated with the central 
government. Traditional units, such as research institutes, universities, and even 
researchers, may to some extent collaborate with the state through applying for 
funding from state-funded foundations or being directly appointed by the state 
to collect data. Social connections play a major role in the allocation of such 
cooperation opportunities (Buckley, Clegg, & Tan, H. 2006; Park & Luo 2001; 
Wong & Tam 2000), which enables researchers with more extensive social and 
political connections to receive more academic research projects from the gov- 
ernment. Another approach of the state is to establish new research centres or 
institutes based on government planning of the political agenda. An example is 
the National Assessment of Education Quality (NAEQ), which was established 
in 2007 to collect nationwide standardised testing data (Zhou et al. submit- 
ted). New testing contrasts with the traditional examination method, which 
has been the dominant means of evaluating, assessing, and ranking students’ 
learning outcomes. However, this new testing carries the ambitious political 
task of diagnosing the “real” education problems with reliable and sophisticated 
empirical data. It is creating a nationwide databank based on the annual assess- 
ment of students’ academic performance and other elements influencing such 
achievement. To accumulate indicators for this new testing system, the state 
has established a special NAEQ centre, employing researchers from top uni- 
versities both in China and from abroad (Zhou et al. submitted). The official 
newsletter of the NAEQ states that more than sixty universities and research 
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institutions are involved in the testing of indicators for different school subjects. 
This testing has been inspired by many large-scale assessments like the Pro- 
gramme for International Student Assessment (PISA) and NEPS (the German 
National Educational Panel Study). 

When we ask what kind of relationship and division of labour exists in China 
between the state and experts and if there is evidence that the relationship is 
changing, we observe that the state tends to rely increasingly on evidence- 
based data. Besides the new testing system, the state is also the main resource 
for research funding. Many research foundations, such as the National Planning 
Office for Philosophy and Social Sciences (NPOPSS), are among the largest state 
foundations. An indication of this increasing tendency is that the NPOPSS has 
started to fund more education projects. In 2015, the NPOPSS funded a total 
of 202 projects (NPOPSS 2016) covering broadly educational topics. In 2011, 
when data collection concerning this began, it funded 167 (NPOPSS 2011). 

The Chinese state channels academic research focus. Government fund- 
ing is usually used by the state as a pre-policymaking test. Research propos- 
als therefore share the concerns of the government’s agenda. The state signals 
the government’s agenda or focuses through documents and leaders’ speeches. 
Newspapers and academics then focus on these topics. For example, one of the 
main concerns of the government is how to solve the imbalanced distribution 
of education and social resources, and QAE’s academic research seeks to take 
this concern into account in using QAE to achieve this. This issue was repeat- 
edly mentioned by various interviewees. It is commonly understood that if 
someone wants to receive government funding for their research project, their 
proposal needs somehow to be connected with the topics prioritised by the 
government agenda. One interviewee stated, 


Our country [refers to the government], for example, now emphasises the 
importance of college entrance examination reform, so we will focus on 
this ...in the college entrance examination reform [of group publications 
related to this issue]. For example, we will ask how to change the college 
entrance examination, how to use the academic achievement test in the 
college entrance examination, how to operate the academic achievement 
test ...and then our focus will be transferred to these issues. 

(CN-E-08) 


Experts play the role of knowledge-broker to inform the state and other stake- 
holders about the situation of education, based on their research results. They 
have connections both with the state in decision-making and with schools 
in implementation. Both connections mean that experts influence decision- 
making through the utilisation of their research results’ evidence and authority. 


[T]hrough this analysis [refers to their project] .. . [our] main pur- 
pose is first to go to help the schools. By analysing the results of the 
teaching process, we hope to help them [refers to teachers and schools] 
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to improve teaching and situations, and their research results are also 
reported to the government for policymaking reference. The second 
purpose is to report the situation to the government so that they can 
learn about situations in schools to reflect on policymaking. Our second 
task is to do policy research to help the government with improvement 
of policy. 

(CN-E-11) 


Experts are largely involved in the consultation stage, but whether the decision- 
maker accepts the results and the extent to which schools can change their 
practice appear to be problematic areas. The state has the final decision con- 
cerning policymaking. A rector of a prestigious research institute told us how 
much their work can influence policymaking: 


The government listens to different voices ... When a researcher delivers 
a report based on their findings, the government starts to hold different 
meetings to listen to voices from different levels. For instance, how parents 
think about or feel about this policy. After many rounds of meetings the 
government publishes the final version of the policy, which looks a bit like 
a decision based on your findings and also a bit like a decision based on 
other researchers’ findings. 

(CN-E 06) 


Policymakers and experts appear to have become a mutual benefit community 
(Han & Ye 2017), which helps and promotes their respective agendas. National 
policymakers have the widest choice of national, local, and international aca- 
demic resources, but they are most likely to cooperate with researchers from 
an equivalent level. For example, the central state usually selects institutes 
based on their reputations and ranking. Likewise, provincial decision-makers 
generally receive academic support from regional universities. Data are more 
likely to be circulated within a political group and among key stakeholders. 
However, policymakers and experts do not really form a community. The 
boundary is clear. The researcher is responsible for data production, while 
the government is responsible for the production of policy based on these 
data: “It is government’s responsibility to decide when and how to release it” 
(CN-E-07). 

In investigating the willingness of Chinese experts and state authorities and 
policymakers to limit and restrict or to widen and open data use in their prac- 
tice of QAE education policies, we observe that national and provincial gov- 
ernments prefer to utilise research results from various sources to support their 
decisions, and even policymakers themselves become experts on data and num- 
bers. These various sources generally emanate from an individual researcher, a 
research group or team, research institutes, and specially funded problem-based 
research centres. The Chinese government has supported prestigious Chinese 
universities in growing their research competence in the global education 
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market through their involvement in international academia in increasing 
impact factors, rankings, and publications. 


The Ministry of Education has multi-channel resources. We have the 
National Bureau of Statistics which is the authority for all types of statis- 
tics. Most documents and policies draw data from this place. The Ministry 
of Education also uses universities, research institutions, and local govern- 
ments to provide data to the Ministry of Education. 

(CN-E-07) 


However, these multi-channel resources only fit the national level. Distribution 
of expertise is uneven. Skilful experts are recruited from national universities 
and serve in national policymaking. At the local level, however, there are fewer 
expert resources. 


There are two skins [two separate sets of actions which have little con- 
nection] in terms of reforms of educational assessment and evaluation. 
The indexes created at national level introduced a new concept of evalua- 
tion, and designed an assessment process ...so the provincial government, 
including the county government, coordinates the assessment. Then the 
results go back to Beijing [the central government] and have no relation- 
ship with locals. So, the locals just continue the old practices, following the 
traditional ways like supervision because they know them well. 
(CN-E-12) 


However, implementation of central policy depends on the regions. Local gov- 
ernment and schools in most Chinese regions are not equipped with equally 
skilled and knowledgeable experts, which may harm implementation and make 
a real difference at the local level. 


The state and expertise in Russia 


Russian experts contribute to international, national, and sub-national discus- 
sions on education and maintain communication between these levels. From the 
state’s perspective, they are legitimate actors responsible for collecting and inter- 
preting data in the field of QAE compulsory education policy. Some worked in 
federal agencies during the 1990s and later moved to the research sector, which 
focuses on measurements and the analysis of their results. Although they are 
autonomous in their international communication, instrument planning, and 
collaboration with the regions, their projects are often stimulated by the state’s 
targeted financing, or they need to look for sub-national partners and funding 
to sustain their research work. 

International contacts ensure awareness of education developments else- 
where. Technical and other innovative ideas are applied in national tests and in 
developing and analysing test items. For example, this occurred when PISA-like 


106 Risto Rinne et al. 


assignments were introduced to the Unified State Exam (USE) after national 
specialists participated in seminars in the Netherlands (RU-E-12). Communi- 
cation with colleagues from abroad stimulates the growth of national exper- 
tise concerning assessments. The interviewed experts insist that this was not a 
unidirectional relationship in the 1990s, since Russian experts were not passive 
recipients of assistance and guidance from international agencies. From an early 
stage they have collaborated internationally on equal terms. 


It is well-known, publicly available, that [international] colleagues did a lot 
for the development and acknowledgement of the Russian instruments. It 
would be wrong not to mention this. However, this is a big collaborative 
work. It is impossible that someone came from abroad, an expert, however 
qualified he is, but it is impossible to come to the territory, country, and 
change something there. It would be wrong. If the team has been formed, 


if there is a constructive moment . . . dialogue, then everything changes, 
you can take the best, and there is a partner-like relationship, it is important. 
(RU-N-02) 


Attaining knowledge on the specifics of the system of education spurred 
the advance of the assessment instruments; later changes included defining 
education standards and development of textbooks. However, the connec- 
tions on this level show the primacy of the state in defining the agenda, 
while experts foster the discussion about the problems of the education 
system. 

(RU-E-04) 


The relationship with sub-national and school-level specialists is more col- 
laborative. The development of new instruments, for example, an assessment 
of children’s readiness for school and subsequent assessments across primary 
schools, occurs through cooperation. This means reorienting schools towards 
development based on available data and giving teachers the opportunity for 
discussion and reflection. In this case, experts involve school specialists in edu- 
cation analysis, and they maintain the link between nationally identified priori- 
ties and practices (RU-E-12). 

However, the federal government is the major player in data production, 
analysis, and dissemination: 


You know, everywhere, always we continued because for the primary 
school we had a project of Rosobrnadzor. It means that there is an organisa- 
tional structure, money, people, plan, results which you aim to achieve. Plus, 
you add something of your own, what you think is important to develop 
in this direction. 


(RU-E-12) 


For example, when Rosobrnadzors support switches to other initiatives, experts 
and school-level specialists have less room for action. They are unable to 
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continue with initiatives they have already started, because the state focuses on 
the new instruments. 

Another example of experts” collaboration at the sub-national level is their 
analysis of the situation of schools facing challenging circumstances. This is a 
matter of current national interest and therefore receives support. Since there 
is less federal regulation, at least currently, there is more room for action for 
experts and school-level specialists (RU-E-04). 

Even when the analysis demonstrates a need for change, there are problems 
in developing practical solutions. Specialists are needed who will collaborate 
with schools and regional governments (RU-E-07). Teachers should be able to 
understand tests and the objective assessment system (RU-E-10). 

Finally, media actors collect data about rankings from open sources. An 
example of such a project is “Social Navigator”, an agency and web resource 
which aggregates information about public services for extensive social use. 
They present their work as being about public accountability rather than the 
pursuit of scientific goals, which is a key difference from the experts’ focus. 
They promote key findings for the information of the public and do not strive 
to influence governance (RU-E-11). 

The relationship and division of labour between the state and experts in 
Russia is built around data collection, ownership, and analysis. Its utilisation 
by different actors varies, depending on their analytical interests and their 
decision-making power in education. The evidence of change is manifested in 
the state’s greater presence at all these stages. The ministry limits experts’ access 
to large-scale data, and decisions are often not based on open professional dis- 
cussion. The state stimulates the development of new instruments, but when 
there is a shift towards new funding priorities, the regions struggle to sustain 
these initiatives. 

Experts connect transnational and national education policies; international 
assessments help experts in legitimising certain ideas on the state level, although 
there is often a delay in implementation: 


I think we always work in advance, this is the feeling of the time-frame 
of our activity, because the expert world is much more saturated with 
information. We travel somewhere, we read, and we get rushed. We want 
models, approaches, and ways of life to work everywhere, to work where 
we are. Then, this time lag emerges, and we feel that nothing happens. And 
if after some time some documents articulating education policies emerge, 
there is a signal from above [informal approval of the state officials], it is 
hard to understand if we weren’t involved. What we did before was to 
create an information field and discussion, because we don’t write just for 
ourselves, you know. It all emerges in the press and becomes a part of the 
public discussion. 

(RU-E-04) 


Typical situations creating political opportunity are presented when perfor- 
mance in some test is particularly low, and the analysis identifies problems 
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(see Piattoeva & Gurova 2018). An example is PISA and the discrepancy 
between its results and those of the Progress in International Reading Literacy 
Study (PIRLS), which has pointed to the weakness of students’ curriculum 
knowledge-transfer to applied tasks (RU-E-12). These cases validate experts’ 
ideas in the eyes of state officials. 

Experts’ influence on the agenda is limited because of the state’s ownership 
of data. Moreover, the interest and approval of state officials determine the 
development and application of any instrument and subsequent data gathering 
and analysis. 

However, the state is also empowered by data, which creates a need for rel- 
evant statistics to support current policy priorities. This stimulates a selective 
approach to statistics, in which only suitable information is extracted to justify 
an agenda, instead of decisions being developed through data analysis. Special- 
ists criticise this case-driven use of inferences which diminishes the demand for 
analytical work. 

However, even when specialists have an opportunity to offer analytical sup- 
port, their expert advice is often unappreciated: 


Look, there is an expert who is asked, he has to respond. I would say it is 
a problem when they [government officials] ask. If they first ask, and then 
you answer, so that someone can take it into consideration, I think this is 
okay. But if something is done, decided, and then they ask you for some 
reason. For instance, there was recently a discussion in the Civic Chamber, 
the ministry presented a model of the graduate portfolio, the portfolio of 
a university graduate. We came there, they did not invite us, we just came. 
We tried to comment, and the answer was “everything is already decided”. 

(RU-E-04) 


Moreover, when cabinet positions change, the pool of trusted experts is re- 
formed. Several experts we interviewed used to work in the governance struc- 
tures of the 1990s, but they are now occupied with analytical work. Other 
specialists are more aligned with governance structures. 

The question of whether Russian experts and state authorities are willing to 
limit, restrict, or widen data use in practising QAE education policies reveals 
acknowledged problems both in data collection and its application in decision- 
making. Although there is an abundance of statistics on many issues, a key data- 
related problem for experts is their uneven collection. For example, context 
data about schools were previously available in the “school social passport”, 
but this instrument was cancelled. Given Rosobrnadzors desire to promote 
evidence-based decisions, experts’ hope is that once a specific data deficiency is 
identified, officials will request a specialist consultation. However, the education 
managers and state officials who order such instruments often fail to understand 
the kind of instruments which might be developed. The cost of undertak- 
ing tests also precludes development in this direction. When large data sets 
are accumulated, sufficient specialists for every school district are required to 
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analyse them (RU-E-02). Monitoring is discussed as the next goal of the edu- 
cation system's transformation, which will assemble an all-encompassing set of 
indicators, but there is a concern about how this data will be used (RU-E-04). 
Although efforts are made to base decisions on them, experts are sceptical about 
how management is exercised in education policy: 


It is called evidence-based decision-making, but it is exotic for our coun- 
try. We don’t have that for sure. The decisions are still being made based 
on opinions: the one who is louder, who has a higher status, who is more 
respected, is the one whose opinion is heard more. In that sense it is good 
that this is already articulated. 

(RU-E-02) 


It is understood that “it is important to note who showed you that piece of 
data”. Its use is often determined by a need to demonstrate something on the 
spot (RU-E-03). Specialists are concerned about how state officials handle data, 
because their insights are frequently ignored (RU-E-03). 

The attitude to rankings explains the difference in the positions of experts 
and education managers. The proponents of this instrument claim it should 
be used for general information purposes as a quick illustration, supported by 
extensive data (RU-N-04; RU-E-05; RU-SN-1). However, experts point to the 
scarcity of publicly available data from which rankings are drawn (RU-E-11). 
Officials acknowledge that rankings do not allow an analysis of the circum- 
stances of poor-performing schools, which is among the governance priori- 
ties (RU-N-04). The instrument does not address challenges in attaining good 
results. Yet results are often used to praise or admonish educational institutions, 
because education managers lack the skills to interpret data. 

There is a general mismatch between the demands of experts, who refine 
the measurements in their scientific work, and state officials, who seek justifica- 
tion for the policy agenda. Specialists exercise autonomy within their analytical 
work, but this depends on government funding and access to data. 


Conclusion: the changing role of the state and room for 
experts in QAE policy in Brazil, China, and Russia 


This chapter’s aim was to analyse and contextualise the changing role of and 
relationship between the state and expertise in Brazil, China, and Russia when 
large-scale quality assurance policies and evaluation in school education are 
undertaken. We examined more closely how the nature of expertise reshapes 
the state’s context, status, potential, and limits in implementing and governing 
education reform. 

As already indicated, the previous research on governance and expertise 
emphasises the state’s changing role, the complexity of national decision- 
making, and a shift to a regime of governance by numbers. This is partly true 
in how both expert communities and the state are willing, at least rhetorically, 


110 Risto Rinne et al. 


to build a community in which shared interests benefit from cooperation. For 
example, in Brazil, we find a strong willingness to cooperate with the INEP 
on different levels. In relation to the INEP’s operational scope, we observe that 
it receives strong legitimisation from different actors. In China, the state also 
tends to rely increasingly on evidence-based data, even to the extent of estab- 
lishing a new testing system. In Russia, independent experts exercise extensive 
autonomy in indirectly influencing education content. The state and experts 
cooperate, with experts providing analytical assistance, while the state owns 
data. The state now demands increased data collection. 

Our analysis confirms that some of the claims presented in previous research 
from a global perspective may slightly exaggerate the situation in Brazil, China, 
and Russia. Expert bodies certainly have some power and, more specifically, 
potential for power. However, the state and expert communities influence the 
whole of society and community. There are tensions between controlling data 
and being controlled by it and who interprets and who publishes data. These 
tensions are more visible in some cases than others. In Brazil, for example, the 
INEP is perceived as quite autonomous, and it enjoys public trust, yet it is not 
entirely free of political influences, which to some extent call into question its 
autonomy. If an expert body’s orientation does not support a political proposal, 
the political process can find a way of circumventing the experts. A similar 
phenomenon can be seen in China. Both national and provincial governments 
have always looked for data and academic research results from various sources 
to support their decisions, but they have always been the final decision-makers 
concerning which and whose data to use. In Russia, although there is an abun- 
dance of figures on many issues, experts’ key data-related problem is its uneven 
collection. For example, context data about schools were once available within 
the “school social passport”, but this was cancelled. Independent experts have 
limited access to state-owned data. The ministry accumulates data to support 
evidence-based policy without extensive professional discussion. The results 
of the USE are unavailable for academic analysis unless they are acquired from 
individual schools (which is problematic for large-scale analysis). At the same 
time, USE results and other data are widely used in rankings. There is therefore 
a scarcity of context data, such as regional socio-economic indicators. These 
deficiencies impede “rule by data”. In Russia, the state can limit and restrict 
data use. 

Policymakers’ and experts’ basic relationship with assessment data use differs. 
Whereas policymakers can work with or without data, experts depend on it. 
Whereas policymakers can bend interpretations, experts attempt to adhere to 
what is analysed. Experts’ independence from state organs is another important 
issue. These differences in understanding data use also reflect the basic dynamic 
of the relationship between the state and experts. 

There is a clear convergence in QAE politics in Brazil, China, and Russia: 
all have striven to expand the collection and use of QAE data by experts. States 
have established new institutions and organisations to widen QAE practice. The 
scale and growth of expert participation in QAE formation and interpretation 
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have been impressive, and many of the characteristics of the “expert state” have 
emerged. However, there is a clear boundary for such expert participation. In 
each country, the division of labour in QAE and the possible limitations and 
restrictions of data use have been discussed. Experts are trusted to produce and 
collect data, but full usage of data is to some extent restricted or interfered with 
by political agendas. 

However, Brazil, China, and Russia differ markedly in their history of col- 
lecting and using QAE data. Brazil has the longest history of institutionalising 
modern QAE data production and dissemination. In China, there has been a 
longstanding and very different tradition of pupil assessment, and the modern 
QAE system is relatively recent. In Russia, the tradition resembles China’s more 
than Brazil's. We conclude that the state’s role in QAE governance is more 
powerful in China and Russia than it is in Brazil. 
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6 Governance by data circulation? 
The production, availability, 
and use of national large-scale 
assessment data 


Nelli Piattoeva, Vera Gorodski Centeno, 
Olli Suominen, and Risto Rinne 


Introduction 


Education quality represents a key discursive justification for diverse ongoing 
education reforms. Using data as a tool of governance (Rose & Miller 1992), 
quality assurance and evaluation (QAE) is a tool for attempting to reinforce 
central control at a distance while allocating more autonomy to local actors 
and simultaneously creating a need for and reliance on new experts and data 
infrastructures (Lawn & Segerholm 2011). Governance at a distance relies on 
the provision and translation of information about subjects, objects, and pro- 
cesses to the centres of calculation and power (Hansen & Flyverbom 2014). In 
this form, the new architecture of governance relies on data production and 
circulation (Ball 2016). 

This chapter starts from the premise that contemporary education govern- 
ance increasingly depends on and utilises quantitative data. However, paraphras- 
ing Beer (2016), our assumption is that it is in the circulation of the outputs 
of measurement systems that we may see the unevenness and adaptability of 
governance at a distance through data, revealing something about the variations 
and messiness of the realities of the broader political orders to which data may 
belong. As Beer (2016) writes, frictionless data circulation is an ideal which 
often remains unfulfilled because of context-specific blockages and frustrations. 
In this chapter, we investigate what we term “the intended political project 
of data”, that is, how education policy documents and statements by officials 
depict where data is supposed to circulate, for what purposes, and how. We then 
look at whether data circulation meets its claimed goals and aspiration by ana- 
lysing data production, data availability, and data use as distinct but very inter- 
twined phases of overall data circulation. In following the scholarly appeal to 
study actual rather than assumed processes of governance by data (e.g., Selwyn 
2016), this orientation advances one of the main issues raised in Chapter 1 and 
elaborated throughout the book, namely, the difficulty of policy-planning and 
the unrealistic promises made in the name of QAE instruments. In doing this, 
we follow the book’s abductive methodological premises to assist in teasing 
out the complex web of possibilities and limitations which exist in conducting 
(new forms of) governance. We thus treat governance attempts as potentially 
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incomplete and riddled with tensions and unfulfilled expectations and seek to 
understand existing governance at a distance better and in greater detail. 

Brazil, China, and Russia have entered the era of education governance by 
data differently and at different moments, bearing somewhat different lega- 
cies of the link between knowledge and decision-making (see Chapter 3). We 
focus our analysis on data produced through one type of National Large-Scale 
Assessment (NLSA) in each country: the SAEB (the Evaluation System of Basic 
Education) in Brazil, the assessments conducted by the NAEQ (the National 
Assessment of Education Quality) in China, and the USE (the Unified State 
Exam) in Russia. Despite differences in the formats of these three assessments 
(see Chapter 7), their comparative analysis is compelling because they share 
three important criteria. First, they represent crucial sources of information for 
national policymaking; second, they are produced and used as quality evaluation 
tools; and third, they aim to make data comparable across space (among self- 
governing sub-national units in Brazil, semi-autonomous sub-national units in 
Russia, and regions with limited autonomy in China) and time (to determine 
changes in learning outcomes). The chapter opens with a theoretical note to 
problematise data circulation as a tool of governance. We utilise the term “data 
circulation” to argue that circulation unfolds through distinct but interlinked 
phases of data production, availability, and use. We see all these phases playing 
a role in the prospects of governance at a distance by data. The chapter then 
moves to our research’s empirical findings, which first address the intended 
political project of data and then the production, availability, and use of data 
explored under the respective sub-titles. The section on the intended political 
project of data uses official policy documents and interviews as empirical evi- 
dence, while the three subsequent sections rely mostly on individual interviews 
with officials and assessment experts. 


Governance at a distance through data circulation 


Scholars use the concept of governance at a distance to refer to modes of 
governance in which formal prescription is complexly absent, indirect, or 
enmeshed with more or less voluntary commitment to accountability, that is, 
to the submission of organisations and individuals to external performance 
measurement, which is often combined with (self)-evaluation (Rinne & Ozga 
2011: p. 66). It relies on those at the centre having information about distant 
persons and events (Miller & Rose 1990). The concept of governance at a 
distance thus emphasises how the behaviour of governed actors is directed by 
the processes of collection and use of information by the authorities, who seek 
to direct the actions and behaviour of those targeted by these data. However, 
this form of governance does not replace but complements what might be 
seen as traditional governance, which has primarily relied on direct prescrip- 
tion (legislation and hierarchical principles absorbed in rules and procedures; 
see Maroy 2009). Various governance styles can therefore be integrated within 
the governance continuum. Researchers thus see governance at a distance as 


Governance by data circulation? 117 


reconciling “decentralized action (subsidiarity, self-responsibility) with central- 
ized assessment (standardization) to facilitate exchange and valuation in vast 
spaces and to make long distance control something the actors aim to achieve 
by pursuing their interests” (Rottenburg & Engle Merry 2015: p. 22). 

Effective governance at a distance necessitates calculations in one place to be 
linked to actions in another not through direct imposition but through assem- 
bling and connecting different actors and agencies in a functioning network. 
These actors, however, may often evince diverse, if not opposite, perceptions of 
and interests in data, reflecting the heterogeneous background networks which 
constitute them. These heterogeneous actors can represent both “producers” 
and “users” of data. To name a few, they comprise the commercial industry 
which produces software and hardware; the highly complex scientific expertise 
which enables the construction of test items and data analysis; policymakers 
who shape the demand for data and its application and set the necessary leg- 
islation which frames data collection, analysis, and use; and the multiple data 
audiences expected to use but which often also need to produce data (for 
example, teachers or officials who first collect data and then use its interpreta- 
tions for various purposes). Government agencies, consultants, auditors, NGOs, 
and ICT personnel, among others, thus need to both supply and apply data 
to generate and consolidate networks as the means and channels of education 
governance (see also Hansen & Miihlen-Schulte 2012; Piattoeva 2015, 2016; 
Hartong 2016). 

We use the notion of circulation to describe how data such as the NLSA 
needs to become integrated in practices, decisions, and processes (Beer 2016). 
Otherwise, data cannot govern (Grek et al. 2011). In other words, the power of 
measurements is located “not just in what they record or calculate, but in what 
then happens to those numbers, how they are used and by whom” (Beer 2016: 
p. 78). Three elements are thus of major importance in data circulation analysis: 
data production, data availability, and data use. Those who order and produce 
data contribute to defining policy problems and solutions; they frame the com- 
plex production chains and the horizon of policy interpretation when data 
are transformed into knowledge. However, researchers emphasise the role of 
the actual availability of data (Souto-Otero & Beneito-Montagut 2016). Data 
should be made accessible to and used by policymakers for evidence - based 
policy and as “a political strategy to make the policy process seem less political 
and more rational” (Steiner-Khamsi 2016: p. 585). Data also need to circulate 
widely across governmental-administrative levels and among different state and 
non-state actors to direct action (Piattoeva 2015). Specifically, they need to 
be employed by local governments and in schools to influence curricula and 
pedagogy through teachers’ and the local administration’s data usage (for this 
aspect see Chapter 8). 

The circulation of numbers can enhance their legitimacy and power if 
they move and accumulate new networks of constituents, technologies, and 
things (Espeland & Sauder 2012: p. 86). The importance of data circulation has 
led researchers to call for a new sociology of numbers and quantification of 
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education which considers how numbers are “mobilized, circulated, consumed 
and contested” (Gorur 2015: p. 13). The flexibility, stability, and combinability 
of numbers, in contrast to the written or spoken word, are said to enable them 
to transcend contexts and find governmental roles in new institutions and often 
for other than their original purposes (see Rose & Miller 1992; Lascoumes & 
Le Galês 2007; Hansen & Miihlen-Schulte 2012). By examining the professed 
political project of data circulation and then looking at the actual produc- 
tion, availability, and use of NLSA data across the three national contexts, this 
chapter contributes to such a sociology of numbers in education, especially the 
debates about actual governance at a distance through data and the realities of 
evidence-based policy and QAE tools. 


The political project of data circulation 
in Brazil, China, and Russia 


Brazilian politics has historically favoured political decentralisation in educa- 
tion, while constantly projecting the construction of a coherent nationwide 
education system (see Chapter 3). The 1990s reforms deepened administrative, 
financial, and pedagogical decentralisation and motivated and justified numeri- 
cal data as the main instrument for obtaining a national perspective of a dispa- 
rate education system (see Kauko et al. 2016). According to the main national 
data expert body, the National Institute for Educational Studies and Research 
(INEP), NLSA data are intended to support policymaking and enable the mon- 
itoring of public policies at the three levels of government (municipal/state/ 
federal), with the main goal of “contributing to the improvement of education 
quality, equity and efficiency. Besides, [NLSAs] seek to offer data and indicators 
about [the] factors [which] influence pupils’ achievements” (INEP n.d.c). 
NLSA data are assumed to circulate freely and widely in Brazil. They 
are considered public data: information which should be disclosed by the 
national provider and which may be freely used and reused, redistributed and 
re-appropriated without legal restriction. As an official from INEP put it, 
“[I]t is not reasonable here in Brazil, with our tradition, to say that you are not 
going to disclose that data” (BR-N-03). NLSA data are designed to “intro- 
duce information to the society” (BR-N-01) and “to produce clear and trust- 
worthy information to managers, researchers, educators and the general public” 
(INEP, n.d.b). Besides this broad public aim, NLSA data are intended to circu- 
late among policymakers working in both the executive and legislative federal 
branches, with the aim of informing education policies (INEP, n.d.b). NLSA 
data should also travel to sub-national units such as state and municipal govern- 
ments, local councils, and schools (INEP, n.d.b). Interviewed policymakers and 
officials stated that NLSA data can and should trigger a vertical chain of reaction, 
resulting in improved teaching practices and education quality. As an official said, 


[The INEP has] the task of collecting the information, and to handle 
that information and to render that information clear to the society. The 
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ministry has the task of taking that information and orienting policies and 
handling policies ... The results that are produced here are often handled 
by teachers on television [the open broadcast TV School] ...and that helps 
teacher training, helps to work in the classroom ... There is an attempt to 
ensure that the teachers and directors should be able to use the results that 
come from these evaluations, that they are able to use [them] to change the 
teaching practice and the practice within the schools. 

(BR-N-02) 


Having closely followed international trends since the late 1970s, Chinese 
scholars started to raise concerns about the shortage of coherent and trust- 
worthy data on the education system's performance. Against this backdrop, the 
Chinese NLSA, the assessments conducted by the National Assessment of Edu- 
cation Quality (NAEQ), was born. An interviewee explained, 


[The NAEQ assessments were established] because there is no data. 
Because the data in Chinese education is largely a mess and there are many 
discrepancies. So, we need data from a place that we can trust. We need to 
get first-hand data. 

(CN-E-02) 


Although the system is still under development, the initial testing phase, which 
officially began in 2007, was phased out in 2015. Documents such as the Out- 
line of China’s National Plan for Medium and Long-term Education Reform 
and Development 2010-2020 (State Council of the People's Republic of 
China 2010) have elevated assessments to serve as the core element of the edu- 
cation system. The purpose of assessments is to understand the system better to 
implement evidence-based policy, which leads to qualitative improvements in 
education (Zhou et al. forthcoming). The previously mentioned reform plan 
for 2010-2020 (State Council 2010) also suggests that it is necessary to establish 
a national information database to develop the quality of education on the local 
level (see also Fan & Liao 2013; Rasmussen & Zou 2014; Zhang & Wan 2017). 
Assessment reforms take place amidst broader attempts to diversify evaluation 
approaches and consequently reduce the number of exam and score-oriented 
practices. Although in the initial experimental phase, NLSA results have been 
distributed only within the government, the plan is to release all the results (but 
not the original data obtained through assessments, a difference which will be 
elaborated on later) to the general public to increase accountability (as of 2017 
no fixed time frame has been set). As one of our interviewees asserted, “I think 
it [releasing the results to the public] serves a purpose. You shouldn’t keep 
data a secret, but it should be published. Only in this manner can we have true 
accountability. Only then can it produce a marked effect” (CN-E-02). 

The Russian USE (Unified State Examination) started as an obligatory grad- 
uation exam for upper secondary schools in 2009, but with growing political 
concern about the unsatisfactory quality of education, a variety of other uses 


120 Nelli Piattoeva et al. 


were attached to it, including informing pedagogy, ensuring accountability, and 
monitoring education quality through the production of quantitative perfor- 
mance measurement data based on exam scores (see Luk’yanova 2012; Tyume- 
neva 2013; Piattoeva 2015). Because these developments transformed the USE 
into a very high-stakes examination for many stakeholders, Russian ministerial 
authorities have recently started to diversify national assessments to relieve pres- 
sure on USE results. However, at the time of the study, the USE remained the 
most important, as well as the longest existing, tool for education quality assess- 
ment, and it is entrenched in the Russian Law on Education. 

The Russian state’s education project is targeted at the unification and stand- 
ardisation of the content and results of national school education. The discourse 
of “putting the state back in”, meaning that the central authorities should have 
more say in the content of school education, is strong and widely supported by 
the authorities and public. National authorities also seek to establish an “effec- 
tive state” system to increase the efficiency of federal regulation and local self- 
governance and bridge the gap between national policy and its implementation. 
However, as the implementation of national school policy is the responsibility 
of the municipalities and depends on their funding, disposition, and expertise, 
assessment data are expected to incentivise local policymakers, school admin- 
istrations, and teachers to understand and target problems in learning achieve- 
ments and introduce changes in line with national priorities. In this respect, the 
initiation of standardised testing runs in tandem with another central policy, the 
introduction of a more standardised, output-based curriculum model in Russia 
(Piattoeva 2015). 

Government programmes for various public sectors, including education, 
embrace detailed, quantitatively articulated objectives and a system of numeri- 
cal performance indicators to monitor the accomplishment of results annually. 
Following this, the results of the USE can be employed as objective perfor- 
mance and quality indicators, for example, in monitoring the quality of higher 
education or the efficiency of local authorities (Piattoeva 2015). Objectivity is 
defined as using standardised, expert-developed evaluation tools as sources of 
impartial information on the functioning of education. Alongside objectivity, 
the ideal of comparability is an underlying value and political goal of the ini- 
tiatives linked with the USE. Comparability is expected to function on differ- 
ent levels: teachers compare their students’ results and simultaneously examine 
the results of other teachers and students in their city, region, or country, and 
the regional comparability sought at the national level (by Rosobrnadzor — the 
National Agency for Control and Supervision in Education, for example). 
Teachers are held accountable for learning outcomes measured by standard- 
ised assessments, and the recent introduction of bonuses and “effective con- 
tracts” has intensified this trend. Teachers’ informed professional judgement 
is highly mistrusted because, it is argued, it is either arbitrary or biased and 
incommensurable beyond its context, and “those who produce the results 
should not be put in a position to evaluate them” (Civic Chamber of the Rus- 
sian Federation 2014). 
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In this section, we have explored the justifications and objectives for the 
NLSA in the three contexts, which illuminate, in CADEP’s terminology, the 
discursive context (political possibilities) within which NLSA data emerge and 
circulate as a tool of governance at a distance. Our analysis makes clear that the 
countries share a discursive space, and all aspire for NLSA data to play a sig- 
nificant role in achieving: 1) the production of evidence for better policy and 
better understanding of the state of education in the country; 2) the closing 
of the policy implementation gap — a goal mainly linked to the vastness and 
internal heterogeneity of the countries and, in the cases of Brazil and Russia, 
to their nature as federal states with substantial autonomies delegated to the 
sub-national level; and 3) the management of education quality (with quality 
bearing a slightly different connotation in each case: see Minina et al. 2018). 
This analysis affords an important background to our exploration by charac- 
terising the discursive context and the national policies to which NLSA data 
are expected to contribute in the three country contexts. In what follows, we 
explore the actual NLSA data circulation. 


The uncertainties of data production 


In Russia, the production of the USE is centralised, being tightly controlled 
by Rosobrnadzor and the technical and analytical units functioning under it 
(for example, the Federal Institute of Pedagogical Measurement in charge of 
developing test items and analysing test scores, and the Federal Testing Centre 
responsible for infrastructure — the distribution and collection of test pack- 
ages). However, local actors like regional information centres are responsible 
for the enactment of exams in strict accordance with national legislation, and 
non-multiple-choice tests are marked by groups of subject specialists usually 
composed of teachers. The national authorities seek to standardise and control 
data collection procedures to the greatest possible extent (through strict and 
detailed legislation and observation, training, and harsh penalties for rule viola- 
tion), while still failing to ensure maximum compliance (see Piattoeva 2016). 
Thus, the quality of the NLSA is a constant focus of political and academic 
discussion, largely because data are inevitably produced by and rely on multiple 
actors with different stakes, and data are therefore expected to play too many 
essentially incompatible roles. 

Problems are caused by the conflicting demands for the USE to become 
a nationwide standardised examination, a tool of accountability, and a source 
of impartial policy — relevant knowledge. In the general climate of increased 
control of the functioning and results of school education in general and assess- 
ments in particular (Piattoeva 2016; Gurova & Piattoeva forthcoming), even 
attempts to produce information on the exigencies of the education system are 
reinterpreted and diverted to the discourse of control: 


There are no groups, except for external experts, who would be interested 
in objective results. For instance, I would be interested in looking at them. 
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But for that we need to refrain from using them for evaluation because 
now the USE carries too many burdens, that is, it is the graduation certifi- 
cate, entrance examination to higher education, and in addition it assesses 
everything possible starting from the teacher. 

(RU-E-03) 


In China, the NAEQ assessments are only starting to take root, and the people 
involved still have little experience of their implementation. Our evidence on 
the topic is therefore limited, especially concerning the actual use of assessment 
data. Nevertheless, our limited research material communicates that there is 
a strong perception of assessment data’s fragility. Assessments are expected to 
counter the practice of using school examination results as the primary indica- 
tor of quality in education (Rasmussen & Zou 2014), stressing the importance 
of providing timely feedback to different stakeholders based on assessment data. 
However, there is a substantial fear among scholars and politicians that new 
assessments may become another high-stakes examination because of China’s 
century-long history of defining education quality in terms of examination 
excellence. The fear is that results will no longer capture the reality, that is, the 
measured and assessed variables. An interviewee explained, 


We don’t want like the schools or students [to] take it as another examina- 
tion. . . . [I]f they think ... they will rank us, then they will do some reac- 
tions, yeah. You know that ... right now ... we get the true answers from 
students. Like I don’t have one hour for my outdoor activities ... but later 
if you take it as high-stake, I am not quite sure they will answer no. Maybe 
the teachers will teach them only to answer yes. 

(CN-E-01) 


The Brazilian academic community frequently contests the methodological 
and political choices underlying the design of the Evaluation System of Basic 
Education (SAEB), maintaining that the SAEB does not enable the identifica- 
tion of schools’ impact on pupils’ learning achievements (for a review of the 
literature see Candido et al. submitted). However, our interview material reveals 
that the INEP’s technical capacity to produce data is rarely contested. It is rather 
the capacity of the SAEB to account for the reality of Brazilian education and 
influence education planning at different levels which is challenged. Whereas 
policymakers, civil servants, evaluation experts, and technicians at the three lev- 
els of government generally stress the necessity of a national standardised evalu- 
ation to achieve education quality, many scholars and practitioners question the 
suitability of quantitative national standardised approaches to assess or make 
sense of educational realities. They consider the Brazilian education system too 
complex to be measured by quantitative indicators, which they characterise as 
superficial. For example, some interviewees raised questions about the signifi- 
cance and method of evaluation by contending its irrelevance either for daily 
school life or Brazilian education as a whole: 
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[T]he pertinence between what is required in these assessments or evalu- 
ation systems, as is the case with the SAEB, and the relationship they have 
with what in fact takes place in the daily life of Brazilian education class- 
rooms, that’s a question that still needs to be better explored in the country. 

(BR-E-02) 


While views diverge concerning the content the SAEB should measure and the 
SAEB itself as an evaluation tool, there is agreement on the inherent logistical 
challenges of data production in such a large and diverse country. The produc- 
tion of SAEB data is centralised. The INEP is responsible for the design of the 
SAEB, although scholars, experts, and teachers are engaged to work on content. 
The INEP hires private companies to collect data and provide certain analyses, 
the results of which are crosschecked with the INEP’s to confirm data valid- 
ity. The INEP works directly with one key person in each State Secretariat for 
Education, whose perceptions about the SAEB and degree of involvement in 
the process seem to affect the smoothness of data collection and the subsequent 
application of results. However, data production remains challenging in a coun- 
try where the main NLSA covers 58,000 public schools and 4,000,000 pupils 
(INEP 2016). The time lapse between data collection, analysis, and dissemina- 
tion does not allow for an in-depth treatment of data because, as an INEP offi- 
cial puts it,“[W]hen you [have] to publish the result, many times you are already 
engaged in another process on another [assessment] to start again” (BR-N-07). 

Quantification relies on a series of interpretative decisions about what to 
quantify and how to categorise and label it (see e.g., Desrosiéres 1998). Pre- 
vious research argues that antithetical attitudes towards data usually emanate 
from high degrees of negotiation of validity and reliability and the tensions 
and dynamics which may be created by methods of data collection (Landahl & 
Lundahl 2013: pp. 57-58). In our cases, these issues also easily conflict with data 
production and explain why actors do not always treat the NLSA as entirely 
objective, precise, or accurate, even when the technical side of producing quan- 
titative assessments is little contested. In Brazil, criticisms of validity are espe- 
cially tangible. In Russia and China, criticism of produced data focuses mainly 
on its reliability and emanates from a mistrust of data production processes. This 
mistrust is linked to political contexts in which data carry high stakes and there 
are historically specific assumptions about the meaning and role of external 
evaluation. 


The ambiguities of data availability 


In China, data generated directly by assessments are only available to the rel- 
evant NAEQ staff and to the highest levels of the Ministry of Education. Out- 
side this circle of trust, the lower government units, the media, the public, and 
even external researchers — as our research team discovered as a result of a 
small misunderstanding in the project’s early stages — are not allowed access to 
data. There is an evident fear that if such unfiltered data spread too far, control 
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of them may be lost and the media will focus on naming and shaming, again 
reviving the high-stakes image of assessments which the government is trying 
to counteract. An interviewee explained, 


In China there is this problem. That is, there is this peculiarity that if you 
publish [assessment] data, they [whoever has access to data] will start mak- 
ing mutual comparisons ... It might be that the consequence is that there 
will be rewards and punishments. But this will cause the objective of the 
evaluation to stray out of line. 

(CN-E-06) 


However, there are some minor exceptions to this rule. As mentioned, the 
assessment system has been developed by relying on foreign expertise. Thus, 
scholars and experts working at the NAEQ have sometimes consulted external 
experts to collaborate on assessment tools. In these cases, national experts can 
choose to provide outsiders with data. During our meeting, an interviewee 
reflected, “For example, I could use our data as an example for you today, but 
it’s absolutely not fine for others to see it” (CN-E-02). 

Despite restrictions of the availability of assessment data, the NAEQ provides 
ready feedback in the form of information or results to sub-national units. An 
interviewee working for the NAEQ stated,“[A]nother division [in the NAEQ] 
is our reporting... division. They use our data to write reports for the govern- 
ments” (CN-E-01). In practice, this means that the scores of a single school or 
the questionnaire answers of a single teacher cannot be extracted from results 
distributed to the sub-national governments. Instead, a county-level govern- 
ment, for example, will only receive information about average performances 
and results in their county and about the national average. They can therefore 
only make comparisons between their own performance and the national aver- 
age or on a time axis by comparing their current and previous performances. 
However, there is some ambiguity concerning whether local governments 
will be allowed to distribute these results to whomever they wish. Moreover, 
in the currently available source material (publicly accessible documents, our 
interviews, and existing research), there is conflicting information regarding 
whether the school level receives any feedback. What is known is that indi- 
vidual students or teachers participating in assessments do not receive it. Again, 
the fear is that assessments will be treated as a high-stakes test. The Chinese 
authorities thus face a situation in which they seek a delicate balance concern- 
ing which assessment data and results are distributed and how. They fear that if 
they are spread too extensively, assessments will become a high-stakes test. An 
interviewee explained, 


So, it [the feedback] is not for the individual. And we don’t want them 
[the students] to get nervous about their individual score. .. . [W]e don’t 
want [the assessment to be] taken as a high-stakes test . . . If you report 


Governance by data circulation? 125 


individuals, maybe . . . the students will get nervous thinking how they are 
going to perform. 
(CN-E-01) 


In Russia, individual test scores of the results and participants of the USE 
are stored in the federal database. The users of the database are said to be all 
the interested federal and local authorities, educational institutions, and other 
organisations involved in the USE. The establishment of the databank, there- 
fore, constitutes an unprecedented achievement — never has the federal state 
possessed comparative data of such geographical scope and detail (see Bakker 
1999). However, evaluation experts argue that the data accessible for research 
are insufficient; the average scores available for public attention, for example, are 
far from being the research data they want or can work with analytically. Thus, 
actors who can perform a scientific analysis of assessment data as a precondition 
for informed political decisions are largely unable to complete the task because 
their access to data is limited: 


The ministry and Rosobrnadzor are keen to accumulate all the information 
for themselves, which they receive from the [regional] centres of quality 
evaluation, from the regional departments and ministries, and it is very dif- 
ficult for society to receive this information. For example, Russia took part 
in TALIS [the OECD’s Teaching and Learning International Survey], you 
know ...So now in order to productively analyse TALIS data, we need 
the USE results of the schools that took part in TALIS. These are only two 
hundred from fourteen or fifteen regions. We cannot obtain this informa- 
tion. Can you imagine? We are the national coordinator, our institute is 
the operator of the project, we don’t need this information for the purpose 
of publicising it in the newspapers ... We need to implement a compre- 
hensive analysis of an international study. Big money was invested. Who is 
interested in the analysis? Who ordered the study? The ministry. 
(RU-E-04) 


Uncertainties concerning data availability surface as a struggle between the 
necessity to collect data for governance and the management of education 
quality and simultaneous realisation that production of performance data 
can easily weaken decision-makers’ immunity and lead to calls for reverse 
accountability. As one Russian respondent eloquently commented, “No one is 
interested in receiving objective information.” He continued that the ministry 
did not wish to see poor results because if poor performance data surfaced, 
the next question would be “What do we need the Ministry of Education 
for?” (RU-E-03). Yet assessment data are widely available in the form of crude 
average scores (of schools, regions, in a particular subject, and so on) as perfor- 
mance indicators employed by officials at different levels of education admin- 
istration and in league tables supported by private and public actors. While 
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authorities frequently state that schools or teachers should not be compared 
or ranked based on testing scores — at least not crudely, with no consideration 
of diverse contextual factors — no strong efforts are undertaken to deal with 
the practice. The national ministry's general message concerning this topic 
remains controversial, as the ministry itself has ordered and refers to league 
tables. 

In Brazil, the situation is the reverse of that in China and Russia. Data dis- 
semination is encouraged by the fact that the INEP produces a large amount 
of varied data, which scholars use extensively. The INEP provides a research 
service and space for consultants, teachers, and researchers to access INEP data- 
bases, the only criteria being that data are used for research or study and ethical 
principles are met. The INEP has also supported several initiatives to meet with 
other education actors to discuss the NLSA and other monitoring tools in the 
context of the National Educational Plan (e.g., INEP 2015). Furthermore, it 
recently created an online platform for teachers to provide them with direct 
and almost individualised pedagogical feedback (data are presented by schools 
and localities, pupils’ proficiency levels, comparisons with similar schools in the 
region, and NLSA items and their pedagogical explanation; see INEP n.d.a). 

Besides initiatives targeting the educational community and relevant politi- 
cal actors, the INEP makes data available to the wider public. It publicly dis- 
seminates the results of the SAEB, which consist of the mean mathematics and 
Portuguese literacy scores combined with several other variables. The INEP’s 
website design and performance has recently been improved, and data are 
extensively available in several formats. Despite these recent developments, all 
the INEP interviewees acknowledged that data dissemination was the INEP’s 
main challenge. Interviewees described INEP reports and databases as provid- 
ing “arid” information which was difficult to understand and recognised that 
civil society organisations were the most important actors in helping with data 
availability and visibility. Even if these organisations frequently draw on data 
to push their own agendas (sometimes their policy proposals are aligned with 
the government’s, sometimes not), their contribution to data dissemination is 
crucial. They are seen to have better communication strategies and display 
results in easily accessible, user-friendly ways. Indeed, local-level interviewees 
frequently referred to the QEdu platform, which is administered by a privately 
funded civil society organisation and extensively and interactively displays 
NLSA data. 

In Brazil, data dissemination is vulnerable to pressure from every sector of 
society, including policymakers, the media, civil society organisations, and even 
school principals, wanting to advertise their place in the rankings (e.g., BR-N-02). 
The media diffuses them extensively. Whenever NLSAs take place, assessment 
processes are scrutinised, technical or procedural mistakes identified, and results 
used to support arguments, usually according to the speaker’s political and edu- 
cation stances. Freely drawing on these results, various parallel polemical rank- 
ings are constructed. Politicians clearly try to channel dissemination according 
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to the political agenda, and professional associations’ and unions’ engagement 
with data greatly contributes to their circulation. 

Despite the shared discourse on rational decision-making and making data 
a prime tool of education governance at various levels, our cases demonstrate 
considerable differences in data availability, that is, who has access to assessment 
data, of what kind, and for what purpose. Here, as in the previous section, we 
see a pattern in which the realisation of shared policy is embedded in the case- 
specific institutional and discursive arrangements and particularities of socio- 
historical dependencies, which can both enable and constrain actual data access. 


Too much or too little data, and 
who knows how to use them? 


All Brazilian interviewees acting nationally, and therefore involved — though 
differently — in national education politics, expressed the view that NLSA data 
have fuelled the debate on education quality. As a representative of a civil soci- 
ety organisation stated, 


[B]y putting the discussion of the metrics based on a large-scale assessment 
you visualise within the education system the inequality [provided by the 
lack of quality] and therefore you make more tangible the talks and the 
formulations and actions about equality. 

(BR-NNGO-05) 


Despite this, the effects of NLSA data on national education policy are dis- 
puted. Several interviewees suggested, as is aptly systematised by a senior civil 
servant working at the Federal Senate, “the consolidation of the evaluation sys- 
tem [i.e., of the SAEB] has been an important driving force of public policies 
in education” (BR-N-11). Other interviewees suggested that the impact is still 
far from what it could and should be. NLSAs collect information which is not 
comprehensively used to formulate concrete education polices which might 
target the improvement of the school as a whole. For many, there is enough 
information to understand which aspects need to be improved in each school, 
but there is no consequential national political use of collected data. 

Several INEP interviewees mentioned that at the state level many govern- 
ments use data carefully in education planning. However, other national, state, 
and municipal interviewees mentioned that most state and municipal adminis- 
trators had little capacity to use data: they were either unqualified to do this or 
they lacked resources. Even in the richer states data use seems to be in its early 
stages. An interviewee from the state level, for example, related that they were 
still trying to “create a culture of data usage”, and that in most states NLSA 
data is still “less used than [.. .] it should be because there are no established 
competencies among the secretaries [of education] to know what to do with 
these results” (BR-S-01). 


128 Nelli Piattoeva et al. 


Nevertheless, some richer states and municipalities have chosen to create 
their own assessments in the guise of preparation for NLSAs and local monitor- 
ing and only rhetorically react to national results. It is argued that NLSAs do 
not fully meet the needs or particularities of the states. However, many states 
and municipalities acknowledge severe financial implications and given the 
existence of the national system, mention only the need to complement it. This 
partial disengagement with NLSAs seems connected to an attempt to ensure 
that education responsibilities are not appropriated by the national government, 
which has used central control in steering certain education aspects, such as the 
establishment ofa common curriculum. Although these initiatives are generally 
welcome, it is obvious that states and municipalities see their own assessments as 
a strategy to retain political authority over education issues. 

While NLSA data met their target in informing education debates and, partly, 
at least, policymaking, they seemed to fail in influencing school practice as the 
political project foresaw. Historically, governmental actions, regardless of level, 
are the object of the education community’s public mistrust. More specifically, 
mistrust of NLSAs emanates from accountability policies. Although neither the 
INEP nor the Ministry of Education supports data use for accountability pur- 
poses (interviewees were unanimous here), some states and municipalities use 
NLSA data to make schools accountable. Furthermore, teachers do not under- 
stand results and do not know how to engage with them. This has triggered 
the construction of the online platforms mentioned in the previous section, 
which aim to inform teachers by explaining NLSAs and translating data into 
classroom reality. However, because of the great number of teachers in Brazil 
and their diverse educational backgrounds, the link between data and classroom 
activities remains a major challenge. NLSA results are still far from being used 
to improve teaching and learning in Brazilian schools. 

In China, utilisation of gathered data is considered a major problem, as prac- 
titioners do not always know how to interpret NLSA data and how to employ 
them in practical solutions. In this opaque context, policymakers sometimes 
choose the feedback that best supports their political agenda, and policy surfing 
means feedback is used to justify a predetermined policy choice rather than 
to identify problems for which policy solutions might then be sought. Once 
the NAEQ has analysed scores and the final assessment results are released, 
the NAEQ staff holds meetings with education officers, inspectors, and school 
principals from different provinces and municipalities. NAEQ experts suggest 
how to tackle problems identified by assessments. These meetings’ empha- 
sis is On cooperative action. An interviewee explained, “The key is in find- 
ing the problems and then helping schools by solving the problems together” 
(CN-E-07). Evaluation experts thus gain in power and become indispensable 
in helping to translate data into potential policy solutions and pedagogical tools. 
However, this also has its limits: feedback does not oblige the recipient to make 
any changes. 

As previous research (Zhang & Wan 2017) and our interviewees point out, 
the assessment system is a somewhat recent innovation. Data use is considered 
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a major problem, the feedback system remains incomplete, and follow-ups 
are rare. Moreover, as our interviews frequently indicated, these new assess- 
ment practices and ideas are still largely emphasised only in official documents, 
whereas in practice, examinations still reign supreme as the primary quality 
indicator. Local practitioners do not always know how to produce solutions 
based on results of a new assessment, even when provided with suggestions. 
A scholar reflected on the general situation: 


I think the challenge is ... how we can use the data well... the results. 
So, I mean, we can do the national assessment. We can easily do all these 
things ... But most important [is] how we can use it ... We want to find 
a way to use results. 

(CN-E-05) 


Sometimes, there is almost an atmosphere of apathy and disinterest concern- 
ing assessments, because they are not always seen as helpful in solving local 
problems: 


For example, tens of counties in Shandong and Henan provinces and in 
Jiangsu province each year receive national-level basic education quality 
monitoring [i.e., they take part in assessments], but they are ignorant of the 
results, and just take care of organising them. 

(CN-E-13) 


Interestingly, the fact that assessments are not used for ranking is partly to blame. 
An interviewee sarcastically noted, 


Some people take it [the feedback] very seriously, but most people don't. 
Because in our education system they care about things that have to do 
with selection, and they are not too concerned about these things that have 
to do with quality monitoring. When would they care? If you used these 
results to compare, they would take notice!” 

(CN-E-06) 


A further challenge is that assessments reveal a significant amount of detailed 
information (teachers’ work burden in one school was an example given by 
an interviewee) beyond levels to which feedback extends; feedback does not 
target the levels at which data were collected in the first place. It appears a 
compromise of sorts has been struck, whereby school principals are informed 
of a problem at a general level so they can attempt to address it themselves. An 
interviewee elaborated, 


So they [the school principals] need to find the causes. But we usually we 
cannot go that deep for them. But we ... will . . . say:‘Okay, this is the phe- 
nomenon we've discovered. Okay, this information we've got’ ... We kind 
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of give them a kind of a direction: “Okay, this is the way you should go to 
try to find out the problem’. 
(CN-E-01) 


In Russia, interviewees often mentioned a conflict between the demand for 
evidence-based decisions on the one hand and the fast pace of performance 
evaluation on the other. National-level civil servants criticised local administra- 
tions for their lack of intellectual investment in the decisions that would bear 
fruit, and their preference for quick fixes (such as dismissal of personnel). 


There are two negative aspects in managerial competences. First, non- 
intellectual reaction, when a head sees a ranking position and says “punish 


the bad ones and reward the good ones”... And second, making deci- 
sions without analysing the causes of problems, or without the adequate 
resources. 

(RU-O-04) 


At the same time, the support provided at the national level to improve mana- 
gerial resources was often limited to simply channelling more data, especially 
comparisons between different regions’ achievements. Analytical support was 
thus supplanted by an increase in the volumes of data distributed during internal 
meetings between the national ministry and regional authorities, for example. 

National experts questioned officials’ capacity to work with data and whether 
data were used for rational decision-making instead of heavy control or one- 
sided reporting. 


The civil servants have no time to pause and think. They work at a very 
fast pace. And if [s/he] didn’t have the required competences to start with, 
it is pointless to hope they will develop. 

(RU-E-03) 


Russian interviewees often complained about the expansion of reporting at 
different governance levels, meaning that, in addition to NLSAs, schools and 
regional authorities spent progressively more time on collecting and channel- 
ling different types of statistical and other information requested and processed 
by higher levels. Interviewees expressed a somewhat contradictory position 
about the simultaneous abundance and shortage of data. Many mentioned that 
data were of the wrong kind. As we discussed earlier, high-stakes data cannot 
be trusted fully even when they are available because data lack the contextual 
information needed for a deeper academic analysis. The difficulty and high cost 
of data collection were also mentioned. In this respect, experts compared Rus- 
sian NLSA data to international large-scale assessments and praised the latter for 
their richness, openness, and depth. 

Concrete understanding of management and data use is still in the mak- 
ing. Some policymakers admit that data cannot be used on their own and that 
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common sense is still needed in decision-making processes, acknowledging the 
limitations of data such as the problems caused by high stakes and the complexity 
of learning outcomes on which the NLSA is intended to reflect (e.g., RU-O-05). 
However, some individual and still rare examples of concrete data use in the 
decision-making process were reported, for example, targeted training in Rus- 
sian language teaching for regions where Russian is the second language, as 
well as the decision to divide the USE mathematics test into basic and advanced 
levels, reflecting the status of the USE as a graduation exam and its recogni- 
tion of the different needs of students in further education (e.g., RU-O-05). 
Surprisingly, despite the frequently cited lack of resources, time, or possibility 
to analyse available data and use it meaningfully for evidence-based policy, the 
motivation to produce more data has remained strong: 


When we receive a set of information on the state of affairs, then in a good 
sense we always have requests for further development. For example, when 
we get a snapshot, we want to see dynamics. When we receive dynamics, 
we want to see the added growth, individual progress. When we know 
the individual progress in maths, for example, we want to have an idea 
about the individual progress in thinking ... And second, [data] should 
be used for decision-making. I want to emphasise again that between data 
and decisions there is a space in the form of the manager’s head, which is 
regularly missing. 


(RU-0-04) 


The idea behind evidence-based policy is that the more information poli- 
cymakers collect, and the faster they collect it, the more informed decisions 
they will be able to make and, thus, the more effective the management of 
social problems will be (Tsoukas 1997). However, this section problematises 
this image of evidence-based policy and points to the complexities of putting 
quantified education data to work in education realities, whether at the policy- 
making level or in regions, schools, and classrooms. 


Conclusion 


Our comparative analysis of the three national cases has explored the diverse 
ways by which data are intended, but often fail or only partly succeed, to be 
integrated in education governance practices at a distance. The political project 
of QAE in general and the NLSA in particular in Brazil, China, and Russia 
envisions NLSA data providing a better general view of the education systems 
of these enormous and heterogeneous countries and a more nuanced view of 
learning achievement at the student, teacher, and school levels, thus optimisti- 
cally and at least rhetorically incorporating very diverse stakeholders in the data 
project. In reality, data are not easily integrated in these actors’ practices, limiting 
their capacity to govern at a distance. Echoing our other chapters, the varia- 
tion and friction the chapter describes reflect the specific political situations 
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and possibilities embedded in the socio-political and historical contexts of 
these countries, such as the various country-specific reforms in education and 
beyond. In Brazil, for example, the strongly democratic open and multi-vocal 
policymaking agenda provides a platform to make performance data fully avail- 
able — a discursive feature absent in China and Russia. Restrictions on data avail- 
ability in China and Russia reflect different contextual factors. They emanate 
from synoptic data possibilities, because in one case governance at a distance 
afforded by data comes at the potential cost of making people in political power 
visible (see Landahl & Lundahl 2013). In another case, there is a fear that pub- 
licly available data may cease to reflect educational reality ‘objectively’. Thus, 
in contrast to Brazil, NLSA data in China and Russia are guarded and used 
exclusively by national officials. Our interviews with civil servants and evalu- 
ation experts often signalled ambiguity and scepticism regarding data analysis 
and use even within official circles. Moreover, restricting data to official usage 
has negative consequences for the overall project of evidence-based policymak- 
ing, as academic experts may also be excluded from the data sources necessary 
for a scientifically sound analysis. 

In all cases, we have detected concern about data objectivity, especially the 
reliability of data collection, which is inevitably performed by actors who are 
more or less external to the evaluation agencies. Data production is problem- 
atic either because it is conducted by locals with varying degrees of profes- 
sional interest in and commitment to the process or because it is generated 
by actors who are then evaluated based on these data. Fragility of trust in 
the processes of data collection is compounded by mistrust of the officials 
and experts involved in administering and interpreting assessments and the 
tensions between experts and officials at different levels of governance and 
between representatives of national and sub-national education administra- 
tion. These complications weaken data, paving the way for further mistrust 
and criticism. Quantification thus highlights the important role of trust 
between civil servants and assessment experts because of the civil servants’ 
reliance on the technical and analytical capacity of the experts (see the case of 
Russia especially). Our analysis equally points to the importance of a work- 
ing relationship between the national and sub-national administration levels, 
especially because of the national level’s reliance on the willingness and pos- 
sibilities of local actors to engage with the NLSA. This relationship seems 
deeply distrustful, rooted in controversies concerning NLSA use as data for 
both evidence-based policy and accountability, as well as attempts to partly 
recentralise certain features of the education system by means of NLSA data 
(especially in Brazil). 

While smooth governance at a distance through data is envisaged because 
of the assumed easy circulation of numerical data, circulation is contingent on 
the successful abstraction and de-contextualisation which allow new mean- 
ings to be attached to data when they are produced for a delimited purpose 
(Porter 1994). Complete or partial failure to achieve an accepted abstraction 
and de-contextualisation impairs or alters the proliferation of networked actors 
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and constellations of new relationships, undermining successful governance by 
data. The simplification of complexity presents an argument against NLSA data 
which is difficult to refute, and various actor segments may make accusations 
of reductionism at any time. As the case of Brazil shows, this kind of criticism 
is made more easily the more data are made publicly available to diverse groups 
of actors. Thus, paradoxically, while data should be made accessible if they are 
to be integrated in diverse practices to influence others” conduct, wider avail- 
ability increases the possibility of questioning data and the diversion of original 
governance intention. In summary, whereas NLSA data are frequently argued 
to offer solid evidence, our study shows that NLSA data can be easily dismissed 
even for mutually controversial reasons. It is either described as too rigid to 
capture complex realities or as too subjective and unreliable to be trusted as an 
accurate and detached description. 

These frictions occur during different data circulation stages and give rise to 
a situation which is characterised not by more data use and reflexivity but by 
the intensification of data production, thus ultimately accentuating a particular 
phase of data circulation. Contrary to David Beer (2016), quoted in this chap- 
ter’s introduction, Anna Tsing (2005) describes friction as essential for motion: 
friction is productive and enabling. Inspired by this alternative perspective, we 
conclude that while data circulation and thus governance by data do not func- 
tion as a well-oiled machine, friction does not slow processes down but instead 
bears a productive power to retune them in new ways. However, as friction 
emanates from historical trajectories and the particularities of political and dis- 
cursive contexts, unexpected directions are neither random nor open to any 
possibility. In our cases, we especially emphasise the politically and historically 
charged dynamics of trust and mistrust in the possibilities of NLSA data, linked 
to the complicated relationships of trust and mistrust between and among dif- 
ferent actors. This situation occurs both nationally and locally and between 
levels and leads to the intensification of data production. For example, in Brazil, 
local actors produce new data to both prepare for the NLSA and generate data 
more suited to local needs. In Russia and China, data production is stimulated 
by problems of data reliability and objectivity. 
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7 Effects of quality assurance 
and evaluation on schools” 
room for action 


Galina Gurova, Helena Candido, and Xingguo Zhou 


Introduction 


This chapter's purpose is to explore the room for action which opens for 
schools with the emergence of quality assurance and evaluation policies. Other 
researchers have studied the relationship between schools and policy. For exam- 
ple, Tyack and Cuban (1995) have analysed policy reform and policy imple- 
mentation in schools; Braun, Maguire & Ball (2010), Braun et al. (2011), and 
Maguire, Braun & Ball (2015) have examined policy enactment in schools; Ball 
and Maroy (2009) have focused on schools” responses to internal and exter- 
nal conditions; and Falabella (2014) has investigated the effects of account- 
ability policies within schools. Previous research indicates that the relationship 
between schools and policy is affected by a multitude of objective and subjec- 
tive variables. Furthermore, “policy .. . cannot be reduced to an algorithm ... 
and the school cannot be reduced to policy” (Ball et al. 2011: p. 637). Finally, 
there are “discretionary spaces” in schools “in and beyond policies”, that is, 
spaces policy does not reach because of actors’ agency (teachers’ good ideas or 
alternative solutions), chance, momentum, or the nature of the object or subject 
under the policy radar (Maguire, Braun & Ball 2015: p. 497). The topic is thus 
far from exhausted. 

This chapter contributes new perspectives to the study of school reforms. 
First, we approach the relationship between policy and schools through the 
lens of governance theories. As demonstrated in the subsequent section and 
in the book's other chapters, quality assurance and evaluation (QAE) acts as a 
means of governance, and in this chapter, we scrutinise the local mechanisms 
of this governance through evaluation. Second, our investigation of schools” 
room for action relies on the analytical framework of CADEP (see Chapters 1 
and 2; also Kauko 2013; Kauko et al. 2012), which prompts us to view schools 
primarily as political actors. CADEP postulates that the key to understanding 
local policy change lies in the analysis of local dynamics: the changing interrela- 
tions, intertwinement with different levels, relations between actors and institu- 
tions, and the main discursive formations and practices. CADEP analyses three 
dimensions: the political situation, the political possibilities, and the political 
room for action. We are primarily concerned with the room for action or the 
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potential of actors to exploit existing situations and possibilities, and we use the 
theoretical frames of organisational analysis and the concepts of economic and 
symbolic capital to explore schools” opportunities (or lack of opportunities) in 
respect of QAE. 

We start with a description of national QAE policies related to school per- 
formance evaluation in Brazil, China, and Russia. These policies create the con- 
ditions in which local education authorities govern schools. We then describe 
local governance mechanisms, which we see as key constitutive elements for 
creating the room for action of schools as political actors. Finally, we analyse 
the opening or restricting of schools’ opportunities in this room for action. The 
following questions guide our investigation: 1) How do local authorities utilise 
QAE in governing schools and why and towards what ends do they use QAE? 
2) What are the opportunities which emerge for schools in relation to local 
authorities’ utilisation of QAE? 

The following sections present the theories which facilitated data analysis 
and interpretation of results; a description of the case localities in Brazil, China, 
and Russia; and a brief overview of national school performance evaluation 
policies and school-level findings. 


QAE in local governance 


QAE has become a strategic governance tool in education politics. Its dynam- 
ics are therefore manifested across the globe at different levels, from the trans- 
national to the local (see Ozga et al. 2011). To analyse how QAE is used in 
local governance, we address three distinctive theories or models of governance 
which stress QAE in specific ways: 1) the “bureaucratic-professional” model of 
governance; 2) new public management; and 3) governance at a distance. We 
briefly describe each in the following paragraphs. 

The “bureaucratic-professional” model of governance (Maroy 2008) refers to 
traditional governance models such as formal communication, labour division, 
hierarchical position, standardisation, and emphasis on qualification, specialisa- 
tion, and professionalisation, which are commonly associated with the use of 
Weber’s idea of efficient and rational processes to organise and maintain the 
social order (e.g., Weber 2015 [1921]). The general claim of traditional bureau- 
cracy is that it is easier to govern rational organisational structures which share 
the same principles as diverse organisations (see also Weber 1949 [1904]). Con- 
formity to general rules and the equality of treatment are emphasised. 

To ensure quality of education in this governance model, the state issues 
norms, rules, and regulations, and controls the compliance of education organi- 
sations and actors to them through such instruments as inspections. Quality of 
education is understood traditionally in terms of teaching quality, which the 
state regulates in cooperation with organised education bodies (for example, 
teacher unions and associations) (Maroy 2008: pp. 15—16). To enhance quality, 
the state organises standardised teacher training and assumes responsibility for 
the provision of sufficient inputs in education institutions. 
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New public management (NPM) is a term coined in the late 1980s to 
address the “new” approach to the management of public organisations which 
is inspired by economic rationalism and business practice (see Hood & Jackson 
1991). Its emergence and propagation were intended to improve public service 
efficiency by applying private sector management models to public organisa- 
tions (see Osborne & Gaebler 1993). NPM applies an entrepreneurial spirit 
to the public sector, reinforces decentralisation, encourages the use of quasi- 
market structures for governance, and emphasises control of outputs. Citizens 
are regarded as consumers and public servants as managers, providers, or sup- 
pliers when corporate governance premises are adopted. These “new” relation- 
ships in the political arena and public sphere are regulated by accountability 
regimes and performance management. The latter comprises performance 
standards and evaluation. 

In education, responsibility for outcomes shifts to education “service provid- 
ers” (schools and teachers), who are guided by national standards. Comparative 
data and instruments such as “best practices” and rankings orient policymaking. 
The state incentivises education providers to improve quality with quasi-market 
mechanisms: the introduction of per capita educational funding, which fosters 
competition between schools, and the implementation of performance-based 
salaries and benefits, which increases competition among school personnel. 
“Consumers” are students and their families, as well as the entire society sup- 
porting public education as a common good through taxes. The NPM model 
of education governance endorses accountability and transparency to make 
“consumers” aware of the quality of the service delivered by the “providers”. 
This is accomplished through QAE mechanisms such as external evaluations 
with publicised results assumed to empower consumer choice. The connection 
of evaluation to performance-based funding and salaries accentuates rational 
NPM ideas and produces a constant feedback cycle in which outputs feed 
inputs and vice versa (for a detailed examination of NPM in general education 
see e.g., Gunter et al. 2016). 

The third governance model which we employ in our analysis of local 
governments’ QAE use also emerged in the 1980s. Governance at a distance, 
as Chapter 6 described in more detail, departs from traditional governance 
methods (legislation, prohibitions, and regulations) and embraces increased 
autonomy and self-responsibility (Kickert 1995). This is enabled by reliance 
on the power of expertise (see Latour 1987) and implies that actors share a 
significant degree of autonomy based on their will to engage in process and 
conduct (Miller & Rose 1990: p. 14). In this paradigm, processes are not always 
prompted by the centre or government, but by an interrelated network of more 
or less autonomous actors. The most obvious advantage of such a governance 
model is that focus on individual agency diminishes the likelihood of resistance 
to governance measures (Kickert 1995). 

QAE procedures contribute to education governance at a distance in sev- 
eral ways. Evaluation schemes produce numerical information, used for com- 
parison and benchmarking, which becomes the key steering tool (Nóvoa & 
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Yariv-Mashal 2003). QAE policies set and enforce standards against which edu- 
cation (education institutions” service, education staff, and education outcomes) 
is measured, legitimising reward and punishment by government authorities. 
Additionally, QAE increasingly introduces self-evaluation practices. While 
these appear permissive and self-initiated, they need to comply with standard- 
ised indicators and central regulations. All these policies and practices implic- 
itly emphasise constant self-regulation and self-improvement (Lawn & Grek 
2012: p. 146). Ball (1993: p. 111) suggests “constraints are replaced by incen- 
tives”, “prescription is replaced by ex-post accountability based upon quality 
or outcome assessments”, and “coercion is replaced by self-steering [under] 
the appearance of autonomy”. Another way QAE enhances governance at a 
distance is provided by education outcomes in a variety of rankings, ratings, 
indicators, and reports. These are publicised and often prepared by the media 
with the purpose of modulating the decisions and actions of students and their 
parents (e.g., Lingard et al. 2016). 


Schools as political actors 


We see schools as political actors because they possess at least some degree of 
political power, which enables them to influence education decisions, policies, 
and outcomes. Thus, our analysis of schools’ room for action is informed by 
the political frame of organisational analysis (Bolman & Deal 2013). This lens 
facilitates the interpretation of local data, since it enhances understanding of 
the ways in which schools utilise QAE policies to gain power and resources. 
Within the political frame, education can be viewed as an ecosystem in which 
schools adjust to external pressures and interact with other ecosystem constitu- 
ents (local education authority bodies, families, and commercial providers of 
education services) to obtain resources. Bolman and Deal (2013) refer to the 
different resources required by any organisation, such as time, money, and atten- 
tion (p. 26) and describe the internal resources connected to an organisation’s 
personnel: people’s skills, attitudes, energy, and commitment (p. 117). 

Bolman and Deal (2013) outline multiple sources from which organisational 
actors can obtain power and influence, which are also necessary in the struggle 
for resources. Identifying such sources is useful to an analysis of the political 
interaction between education authorities and schools. For example, local gov- 
ernment authorities can be viewed as possessing a coercive power based on the 
ability to legitimately constrain, prohibit, interfere, or punish. Local authorities’ 
power may also be based on the control of rewards — the ability to deliver jobs, 
money, and political support. Concomitantly, schools may draw influence from 
other sources, in particular, from their expertise and reputation, alliances and 
networks, and control of agendas (for a more detailed description of sources of 
power see Bolman & Deal 2013: pp. 225—242). 

To broaden the perspective of the resources schools can gain through QAE 
policies, we use the concept of material and symbolic benefits, which stems 
from the distinction between economic and symbolic capital (see Bourdieu 
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1984 [1979]; Bourdieu 1998 [1994]). Bourdieu (2013 [1980]) argues that eco- 
nomic and symbolic capital are inextricably combined in the social space; 
one complements and reinforces the benefits of the other. The material ben- 
efits schools can obtain through QAE policies are easier to track and meas- 
ure, because in most situations, they are connected to government budgets or 
domestic aid programmes. At the same time, not only do individual interests lie 
in the material order, but individuals and organisations aim to enhance sym- 
bolic dispositions, such as prestige, status, and honour, in acting in the social 
space (Pinto 2000). The value of symbolic benefits arises from the recognition 
(perception, understanding, and assurance) of relationships established between 
those who possess and utilise symbolic capital (Pinto 2000). Symbolic benefits 
can be obtained from a wide constellation of sources. For example, the achieve- 
ment of a top-ranking position in league tables or awareness (through advertis- 
ing or other marketing campaigns) of good scores in standardised evaluations 
can afford valuable symbolic benefits to schools. 


Case localities 


Our data were collected in selected localities in each of the case countries 
through document analysis, interviews, and observations (details on data collec- 
tion and analysis are provided in Chapter 2). Given the data collection meth- 
odology, our findings are not representative of the countries or localities in 
which our research was undertaken. We aim to reveal the diverse mechanisms 
of QAE policies’ influence and to understand the logic and interests of schools 
concerning performance evaluation mechanisms. However, when analysed 
comparatively, the perspectives of local education practitioners reveal similar 
patterns which are arguably meaningful for a more general analysis of schools’ 
room for action. 

In Brazil, the data were collected in the southern state of Santa Catarina 
(population 6.4 million). The state’s social indicators are among the highest in 
the country and across Latin America. Its prosperity derives from its diversi- 
fied and industrialised economy. We selected Santa Catarina for our research 
because it has been especially active in the introduction of QAE policies. It is 
the only Brazilian state to order a report from the Organisation for Economic 
Cooperation and Development (OECD), and the 2010 Programme for Inter- 
national Student Assessment (PISA) recognised Santa Catarina as the leader 
in Brazilian QAE. Since 2005, Santa Catarina has attained the highest IDEB 
(Index of Basic Education Development) of all Brazilian states. Interviews and 
observations were conducted in three large public schools (each with around a 
thousand students and fifty teachers) and in key state and municipal education 
organisations. 

In Russia, we conducted our local study in the Republic of Chuvashia 
(population 1.3 million), which is approximately 650 kilometres from Mos- 
cow. Chuvashia is representative of mid-size, middle-income regions, with 
about half its population of non-Russian ethnicities. The region has a 
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well-developed QAE system which has received positive reviews from exter- 
nal evaluators and is often presented as an example of best practice at training 
sessions for QAE professionals (Bochenkov 2013). The World Bank guided 
the education reforms implemented in the republic. Our local case from Chu- 
vashia is Cheboksary, a city of half a million and the capital of the republic, 
as well as the centre from which all regional QAE initiatives originate. We 
collected this chapter’s interview and observation data primarily from two 
public schools in the same city district, as well as from municipal education 
organisations. 

Local data collection in China followed a different pattern because of schools’ 
restricted accessibility. Intensive school observations were not allowed, so our 
analysis relies solely on data from interviews with school principals and deputy 
principals conducted in seven schools from two provinces in Northern China. 
One province belongs to a well-developed region. New education reforms are 
being piloted and tested in the locality, making it one of the most advanced 
areas in adopting national QAE policies. The other, situated on the coast, can 
be described as moderately developed, with an income level slightly above the 
Chinese average. Although it has a reputation for producing competitive stu- 
dents, it is neither labelled nor known in China as the most active or pioneering 
in innovating and implementing education reform. While the process of QAE 
policy introduction varies across different provinces, here the policy has been 
adopted moderately quickly. The schools visited in the two provinces varied 
greatly in their history, ranking positions, and student bodies. We conducted 
interviews at different types of schools with the goal of obtaining greater diver- 
sity in interviewees’ positions and opinions. 


The rise of school performance evaluation 
in Brazil, China, and Russia 


In each country, new evaluation instruments measuring students’, schools’, and 
teachers’ performance have recently been added to the traditional QAE system 
(Chapter 3 discussed the development of national QAE systems in detail). In 
Brazil, such performance evaluation instruments are the SAEB (Basic Education 
Evaluation System) and the IDEB (Index of Basic Education Development); in 
Russia, they are the national examinations (GIA), which assess students’ educa- 
tion outcomes after grades 9 and 11; in China, they are the NAEQ (National 
Assessment of Education Quality), which assesses students’ academic achieve- 
ment in grades 4 and 8. 

Education statistics — or the “school census” — in Brazilian education began 
in 1931. The collection of statistics was gradually decentralised to the Brazil- 
ian states, which then sent a compiled data set to the federal government. The 
SAEB was introduced in 1990. It consists of two principal biannual assessments 
of maths (problem solving) and Portuguese (reading), one involving a sample of 
pupils in both primary and secondary schools, the other applying to all pupils 
in public schools registered in grades 5 and 9 and popularly known as Prova 
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Brasil. SAEB results are used in calculating the IDEB, alongside school flow data 
(progression, retention, and dropout rates) provided by the school census. This 
indicator was created in 2007 to measure the quality of each school, municipal- 
ity, and state and the overall quality of national education. 

The SAEB is claimed to contribute to the improvement of education qual- 
ity and the universalisation of school access by subsidising policy formulation, 
reformulation, and monitoring (INEP 2016). It is intended to enable a bet- 
ter understanding of the variables which influence pupils’ performance. The 
SAEB is also intended to increase participation of parents and society in educa- 
tion (Brasil 1988, 1996, 2007, 2014). Finally, researchers envision the SAEB as 
an instrument to enhance local education management at sub-national levels 
(Machado & Alavarse 2014). 

The SAEB and IDEB are national policies. However, Brazilian states and 
municipalities, which are the country’s major providers of public education, 
have the autonomy to develop their own student performance evaluations, and 
use them differently for simple comparison, diagnosis, school and staff accredi- 
tation, and performance-based remuneration. Around twenty states (Brooke, 
Cunha, & Faleiros 2011) and more than 1,500 municipalities (Bauer, Pimenta, 
Horta Neto, & Sousa 2015) coordinate a standardised evaluation in addition 
to the SAEB. All evaluation mechanisms in primary and secondary educa- 
tion, except for university entrance examinations (vestibular and ENEM), are 
low-stakes. 

In 2007, however, the school census started to investigate student and teacher 
data, comprising data about school infrastructure, docents, enrolment, school 
hours, and school flow by level, stage, and type of education. The school census 
is a reference for calculating public school funding as well as for managing sev- 
eral federal programmes. Besides being responsible for the collection of these 
reports, the principals of several Brazilian public schools are also accountable for 
the fulfilment of action plans derived from management projects they presented 
when they were elected. 

In China, education supervision comprising administrative and education 
inspections plays a major role in school quality assurance as a sub-system which 
complements local education governance. The supervision service collects sta- 
tistical information on schools and ensures that education policies and plans are 
implemented by schools as expected. Local governments provide schools with 
funding, salaries, in-service training, and promotion for school staff based on 
this information. 

In 2007, China introduced a new national assessment of education quality 
(NAEQ), employing standardised testing as a supplement to education supervi- 
sion, which resembles international large-scale assessments, such as PISA, and 
Trends in International Mathematics and Science Study (TIMSS) (NAEQ 
2015). The NAEQ evaluates students’ achievement in Chinese, mathemat- 
ics, and science. It also includes indicators of students’ physical and mental 
health and socio-economic conditions. Assessment results are accumulated in a 
national database to inform policymaking. Performance data are also circulated 
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among local education actors. The report is delivered by the NAEQ’s national 
centre to avoid different provinces or localities competing with each other, 
and it aims to provide comprehensive information to policymakers so they can 
learn the real level of educational development and problems in schools. The 
performance report is sample-based, and since 2015, all Chinese provinces and 
municipalities have received an annual assessment. 

As China has a long history of ranking the quality of education with exami- 
nation scores or other administrative methods, this new model of testing, along 
with the popularisation of student-centred pedagogy, is expected to use qual- 
ity evaluation to reduce ranking and ruthless competition in schools. China’s 
ten-year blueprint for education development between 2010 and 2020 (State 
Council 2010) outlines the central government’s intention to renew and 
upgrade evaluation practice at all levels. Nevertheless, at the local level, school 
quality is largely defined by students’ examination scores, which determine 
their opportunities to enter the next education levels. The scores thus remain 
parents’ primary concern. 

In Russia, the traditional instruments of quality control in school education 
are inspections and school reports. Until 2009, school-leaving examinations 
were conducted by each school, supervised by the local education authorities. 
Examination, end-of-quarter, and end-of-year grades served as the main indi- 
cator of students’ educational achievement. Policymaking was also informed 
by statistical information on school facilities, teacher qualifications, and student 
numbers. These data and grades assigned by schools are still included as indica- 
tors in the quality evaluation system. However, they are complemented by data 
from diverse large-scale assessments of student achievement and largely from 
national examinations. 

National examinations were introduced in the 2000s as a key element of the 
newly developed system of evaluation and quality control in education. Their 
introduction was a way to ensure the “unity of educational space” across the 
diverse country, and at the same time, these standardised tests were to produce 
statistics on education quality for more informed policymaking. Finally, at the 
time of its introduction, the national test was acknowledged as a strong measure 
of equality and quality improvement (Bolotov 2004). 

The Unified State Exam (USE, or GIA-11) is sat on completion of the 
eleven-grade school education, and the State Final Attestation (SFA, or GIA-9) 
is undertaken in grade 9, the last grade of general education. Examinations test 
graduates’ knowledge in two compulsory subjects (Russian and mathematics) 
and several chosen subjects. Passing the tests is necessary to obtain a gradua- 
tion certificate and apply for the next level of education. Contents of the tests 
are developed independently of schools in connection with the compulsory 
state curriculum. Scores in both tests serve as the main indicators of education 
quality in national and regional policy documents on quality assurance, school 
rankings, teacher performance evaluation metrics, and even (until 2015) assess- 
ments of regional administrations’ effectiveness (Piattoeva 2015). Thus, national 
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examinations carry high stakes for all involved im education: students and par- 
ents, school workers, and the education authorities. 


Local governance of schools through 
quality evaluation and assurance 


The room for action of schools as political actors is conditioned by local edu- 
cation authorities’ use of evaluation procedures. In each of our case localities, 
diverse control and governance measures are applied to schools based on perfor- 
mance. In considering the whole range of traditional and new QAE procedures 
implemented in each locality, we focus on developments in local governance 
models inspired by the introduction of performance evaluation instruments. 
Our data analysis was informed by the three theoretical models of governance 
described at the beginning of this chapter; our interviewees’ specific accounts 
or other indicators in our research data caught our attention, because they were 
characteristic of a particular governance model. At the same time, we sought to 
reflect the complexity of local governance realities in our findings, not limiting 
them to models but enriching the understanding provided by a combination of 
the three theoretical devices. 

The governance practices this section describes are those mentioned by our 
respondents, presented in the local policy documents we analysed, or which we 
observed in the localities. Hence, “Brazil”, “Russia”, and “China” in this sec- 
tion simply label the source of data and should not be taken to imply that the 
attributed governance practices are consistent across the countries. 


Evaluation as an indirect intervention 


In accordance with national discourses on evaluation and quality improvement, 
by introducing new evaluation procedures, local authorities seek to change school 
management and teaching methods without direct interference. Evaluation is 
regarded as a means of reorienting school staff to different aims more aligned 
with national education priorities and encouraging school management and 
teachers to devote extra effort to improvement. In all observed localities, schools 
were required to regularly produce specific documents in which they declared 
their goals and evaluated their progress (in Brazil this was labelled “action plans”; 
in China, “self-regulation”; and in Russia, “self-evaluation reports” and “plans for 
the improvement of the effectiveness of learning”). In other words, local author- 
ities use QAE procedures to ensure and stimulate schools’ self-improvement, 
based on the assumption that schools would probably not do this on their own. 


Both the internal evaluation process, which requires an action plan, and 
the external evaluation process should demand action plans in order to 
overcome the appointed limitations and difficulties ... We want to make 
schools commit to start discussing evaluation, discussing self-evaluation, 
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its own resources, what is more important, what is better, ideal, because if 
nobody discusses anything, nothing will be taken into account, and hardly 
anybody will do it. School management, for instance, has a huge commit- 
ment and in the end, induces this. In case it doesn’t, everything gets very 
loose ... And that's how the human being is... The teacher lets the routine 
get loose, students also get loose ... [when the teacher] does not demand, 
does not require people to take some position ...— then we know [what 
happens — a pessimistic scenario results]. 

(BR-S-11) 


Attention is paid to school staff's ability to work with data. By training teachers 
to produce and analyse evaluation data, authorities expect them to become bet- 
ter informed of their students’ learning gaps and to close them more successfully: 


A teacher should be able to work with feedback, which means to do tests, 
to encourage advanced learning ... The problem is how to know, or, rather, 
how to influence what happens in the classroom, because the teacher needs 
to know what he or she is teaching. 

(RU-M-2) 


Stimulation of the production and management of data in schools is intended 
to improve staff's reflexivity and critical thinking and direct their attention to 
problems the authorities deem important. For example, a Russian interviewee 
said that one of the aims of evaluation was to make schools pay more atten- 
tion to low-performing students (RU-M-2). In China, it is hoped that new 
evaluation will loosen the current overarching emphasis on examination results 
and redirect attention to students’ well-being in schools and in their general 
learning context (State Council 2010). Evaluation is perceived not simply as 
a new tool but as a new way of thinking, an “evaluation culture” (BR-S-11), 
which should permeate schools’ main practices, from planning and managing 
to classroom practice. 


We have documents and bulletins that are specific to the principal, which 
give him a different view in relation to these indicators. We have bulletins 
for teachers, and there is one that students take home to their parents ... 
We invite 150 professionals to participate in workshops because we want 
them to be able to get all the knowledge, not only know how to read a 
Prova Brasil result, but to make them qualified to do a specific analysis of 
the results, and willing to know the indicators ... Our main objective is 
to strengthen this view, to enlarge this view and to make them gain more 
benefit from what is available today. 

(BR-M-2) 


We have never thought that context indicators can be used, for example, 
to introduce some new things, to form new administrative structure in 
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schools, for example. [In schools] they are already interested to ensure that 
information is collected, so a [special] deputy principal is appointed who 
supervises evaluations ... Also new deputy principals for primary educa- 
tion are appearing. Previously, not all schools had them, but now there are 
all-Russia tests [vserossiiskie proverochnie raboty, national tests at the primary 
level], so there is a new [administrative position]. 


(RU-M-2) 


Some of our interviewees from schools in Brazil and Russia expressed scepti- 
cism of the authorities’ declared intention to foster improvement in schools. 
They regarded the “improvement through evaluation” discourse as a disguise 
for the authorities’ real intention, which was to shift all responsibility for quality 
to schools and teachers. Numbers inadequately represented quality of educa- 
tion, school actors explained, and if the authorities were sincerely concerned 
about quality and the situation in schools, they would “come and see” instead 
of calculating indexes. This view was shared by an interviewee from the Rus- 
sian local authorities: 


When schools submit reports, [numerical] analysis, we still look at ... what 
problems reveal themselves [in these reports]. We understand that we need 
to go and visit these educational institutions, we should see and help them 
in this respect ... After national examinations we worked through all [iden- 
tified problematic] issues with every [school] administration, we created a 
whole schedule of meetings with administrations, yes, we worked overtime 
for a week with our schools, discussed where the gaps are in those institu- 
tions, what to do to eliminate the gaps. 

(RU-M-1) 


This illustrates that authorities question the adequacy of data and numerical 
evaluation tools for quality assurance. The next section explains in more detail 
how evaluation is connected to other governance instruments. 


Justification of reward and punishment 


School inspections based on performance evaluation were common in our 
observed localities in China and Russia and are under discussion in Brazil. 
The functions of such inspections include identifying problems contributing 
to unsatisfactory school performance, demanding schools develop plans for 
improvement, and in some cases, applying sanctions. In Russia, inspections are 
authorised to issue fines and even suspend school licences. In China, schools’ 
failure to meet performance standards may also have severe consequences. 
A first failure means the school will be disqualified from annual rewards, and 
a principal’s promotion may be deferred. Teachers may lose the opportunity 
to pass their annual appraisal and gain promotion. If a school continues to fail 
within a given time, it may be suspended or closed. However, such extreme 
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decisions are very rare and usually apply to “low-quality” schools. In public 
schools in both China and Russia, the principal’s suspension is another poten- 
tial consequence if the school consistently underperforms. Although such harsh 
measures prescribed by inspectors are relatively rare, the possibility of such 
measures places great pressure on schools to avoid poor results in performance 
evaluation. Inspection itself can function as a form of punishment or threat, 
even when high stakes for school administration are not attached to it, which 
we witnessed in Brazil. 


Respondent: If there are some problematic issues, they need to be identified, to 
be demonstrated. 

Interviewer: | And what support was offered to schools that had many problems? 

Respondent: Well, there were not many of them, in practice ... In some, the 
directorate was changed (RU-M-2). 


Nobody wants to receive the external commission, which is a commission 
created by the [sub-national authority] that goes to the school to check 
what led to a low IDEB in that school ... This process will induce every- 
one to grow, everyone to increase [IDEB], always, because nobody wants 
to have the thirty lowest [IDEB scores and be inspected]. 

(BR-S-11) 


Performance evaluation also facilitates governance through diverse mechanisms 
which link evaluation results to provision of resources. First, evaluation can 
justify distribution of funds, including performance-based funding and salaries, 
project funding, and grants. Salaries of Russian school administrators and teach- 
ers contain a performance-based element. In the Russian case, locality students’ 
GIA results are also considered in school and teacher contests and as criteria for 
participation in special projects involving extra funding. 


We had an indicator connected to student absence in schools. Currently 
we still have it, but then we made a specific emphasis on it, we made 
[schools] watch these numbers more attentively. It was our goal to stimu- 
late them with this criterion so that they would improve the situation. For 
example, if her or his salary depends on the improvement of this indicator, 
then ... there is a motivation for the manager to work on it. 

(RU-M-2) 


A Russian school principal reports, 


The municipal Department of Education demands a certain percentage 
[of high grades which the school students need to obtain]. If we do not 
provide the required quality, it means that we produce pedagogical defects. 
Our funding is decreased then. ... We could not even apply for some 
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grants, because only schools without students with a criminal record could 
apply, and we have such students. 
(RU-S-2) 


In Brazil, high performance is not the only route to resources, as there are 
also national and sub-national government programmes for providing low- 
performing schools with financial and technical support. Bonus funding to 
high-performing schools in some Brazilian regions is provided to reward 
staff performance, and some schemes function under which high-performing 
schools become sponsors and advisors of low-performing schools to improve 
the local education system’s overall performance. In China, different localities 
have different traditions of encouraging schools to achieve better results in 
college entrance examinations. An interviewee reported of certain localities, 
“They give a bonus to some schools depending on how many students are 
recruited by Qinghua University or Peking University, or how many students 
are recruited by top universities” (CN-P-01). 

In Russia and China, performance evaluation results count in awarding qualifi- 
cations and honoured status to administrators and teachers. In Russia, professional 
qualification and status influence salary levels and are also crucial for professional 
recognition. In China, they are not connected to remuneration, but as symbolic 
rewards, they are highly valued, as a Chinese school inspector explained, 


The punishment of a teacher who, for example, failed the moral evaluation, 
is that he or she cannot participate in teacher promotion in this year; he 
or she would not participate in the selection of honoured teachers. This is 
quite a severe punishment for a teacher. Think about that, if as a teacher 
you could not get promoted or become an honoured teacher because of a 
moral problem, how can you gain trust from students and parents? 
(CN-S-05) 


In all three case countries, low-performing schools are offered support in the 
form of supervision and training for teachers or peer assistance in teaching. 
However, some interviewees from schools regarded this practice not as support 
for teaching methods but as symbolic punishment, which labelled such schools 
and teachers as incompetent. 


They [the local authorities] came up with a very “interesting” project. For 
example, one subject teacher in our school got six fails in GIA results [six 
students from his/her class failed in the subject examination]. And this class 
is to be visited by another teacher who had no fails, so that he/she can 
conduct the preparation-for-GIA lessons with the students of this class. 
What is the implementation of this project going to demonstrate? That 
this teacher [the one who had fails] did not do his/her work, and another 
lady now comes — oh so smart, look, children! So, what image of this class’s 
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teacher is this project going to create? Do they think of it at all? And this 

[intervention by a different teacher] is not going to work, in just one week, 

because students do most for the teachers they know and love. 
(RU-SM-4) 


The practice of using performance evaluation as justification for reward and 
punishment does not always mean that authorities implement evidence-based 
policy. Sometimes decisions about reward or punishment come before evalu- 
ation results, which are subsequently used as justification for decisions. This 
mainly concerns evaluations in the form of inspections but is sometimes 
applied to numerical performance evaluations. A Russian interviewee told us 
that a ranking based on numerical indicators was considered inaccurate by a 
supervising authority because it contradicted the authority's opinion of who 
should occupy the top of the ranking: 


Last year we tried to rank administrators based on the indicators given by 
the [regional] Ministry [of Education] ... This was... even a small argu- 
ment, you couldn't call this a conflict, but just an argument about “why you 
have chosen the wrong school [as ranking leaders]. 

(RU-M-2) 


In Brazil, some school interviewees perceived that evaluation criteria had been 
developed to accommodate political interests and produce better results. 


I see that the main role of external evaluation is to manipulate indexes, 
manipulate a situation so public schools look as if they ... have good 
conditions. I see that there is a clear lowering of education [standards] in 
public schools, and this [is] veiled [by politicians, otherwise they will not 
be elected] ... That's when the index is essential. 

(BR-MS-1) 


Accountability of authorities 


Local authorities themselves are subject to evaluation and sanctions because 
they occupy an intermediary position between schools and the sub-national 
and national levels. Local officials need to demonstrate that they are success- 
fully implementing national and sub-national regulations and doing something 
about schools’ identified problems. In other words, local authorities undertake 
evaluations to report to their own supervisors. 


Since 2009 we have established a notification system of supervision results. 
The results [are] included as one of the main indicators of the county 
government performance to offer evidence for reward or punishment ... 
Moreover, the problems that have been found in a previous inspection will 
be put into the special checking list [for] next year. 

(CN-M-02) 
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It is therefore in local officials’ best interest to ensure that schools pass inspec- 
tions. A Russian interviewee reported, 


Supervision authorities visit [schools] quite often, be it [the] prosecutor’s 
office or the Department for Supervision and Control in Education. They 
take a certain aspect and review it. But before they visit I have already assigned 
a specialist for this task; she goes in advance and, together with the school, 
checks all the documents. [She] also observes lessons, because sometimes 
there are [federal] tests in different subjects. She, so to say, prepares schools for 
these inspections. And when the Department for Supervision and Control 
comes, she is there with them and polishes (otrabatyvaet) certain issues. 
(RU-M-1) 


A Chinese respondent explained that the need for local inspections arose from 
other evaluations: 


Lots of evaluation of schools is still based on school [graduates] per- 
formance in college entrance examination results. Schools might go to 
another extreme to [devote] all the time to the examined subjects. Many 
schools don’t even teach the subjects not tested. So, supervision makes sure 
that no such actions happen in schools. 

(CN-S-01) 


With the lack of supervision and inspection in Brazil, sub-national govern- 
ments tend to use teacher training to ensure the curriculum has been followed 
and the evaluation culture has spread. 

Local authorities are also accountable to the public, especially parents, so 
authorities use QAE instruments to demonstrate their work and schools’ qual- 
ity to the local community. At the same time, evaluation serves as a means by 
which authorities involve community actors in the governance or support of 
schools. For example, both Brazilian legislation and public opinion call for the 
participation of community and families in education and, in a context of high 
social inequality, voluntary work and donations often target low-performing 
schools. The authorities thus view evaluation as a tool to provide relevant 
information on schools to the community; evaluation needs “to reflect the real- 
ity of education ... [and] raise [the] involvement [of community members in] 
schools” (BR-S-9). In Russia, some local quality assurance measures, including 
inspections, are implemented in response to parents’ complaints. Self-evaluation 
reports prepared by Russian schools are also published to provide parents and 
the public with information about school quality. Chinese interviewees also 
mentioned that evaluation and inspection in some localities involve parents, 
the district community, and the media: “We [inspectors] interview students 
and parents, and then the community around the school. We also view the 
school archives, and then assess the implementation of rectification, [and] do [a] 
follow-up investigation” (CN-M-01). 
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Schools” room for action in QAE 


Local authorities’ use of QAE in Brazil, China, and Russia to govern schools 
sets schools’ room for action in their localities. Remembering the aims and 
actions of authorities described in the previous section, we now proceed to 
an analysis of school administrators’ and teachers’ reactions. In this section, we 
seek to answer our second research question: what are the opportunities which 
emerge for schools in relation to local authorities’ utilisation of QAE? We 
look at schools’ opportunities as political actors — their opportunities to obtain 
power and resources. For this analysis, we employ the typology of the sources of 
power (Bolman & Deal 2013), which emphasises the diversity of such sources 
(or kinds of power). We consider not only material but also the symbolic 
resources schools can obtain, and focus on both those who gain and those who 
lose power and resources because of changing QAE policies. 


Evaluation as an internal management tool 


Some school administrators in Brazil and Russia eagerly embrace evaluation 
as an effective management instrument. They see comparison as a “natural 
way of human thinking” (RU-S-3 and BR-SS1) and use both traditional and 
new evaluation tools to encourage their teachers, students, and parents to work 
harder to get better results. Schools can refer to evaluation data at staff meetings 
to set goals for the upcoming period, identify student groups requiring extra 
attention, and praise high-performing teachers and scold low-performers. One 
of the Russian schools we observed organised an internal ranking of teachers 
based on their students’ performance and regularly designed contests for stu- 
dents and staff members. 


The internal quality control [vautrishkol’nyi kontrol’, a traditional proce- 
dure involving regular lesson observations and peer discussions] is the main 
thing! I always tell deputy principals: you can postpone anything, but do 
the internal control! One should reveal a problem in time and solve it 
quickly, then you will have good quality in the end. For example, in one 
grade 8 group students suddenly started getting fail marks. It turned out 
that their former teacher worked insufficiently, and the deputy principal 
didn’t find it out in time. So, I went teaching ... that group, we cleaned a 
little bit [“cleaning” means getting rid of low-performing students, primar- 
ily by persuading them to transfer to another school], and in grade 9 they 
passed national examinations more or less satisfactorily, though there were 
still five fails in that ... group. 

(RU-S-1) 


When I see the IDEB, the first thing I do is to compare my school with 
others — I use all data. . . . I compare with other schools... [in the neigh- 
bourhood] in order to understand, because the region has similar socio- 
economic characteristics, so we can compare. This is natural and for me, 
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indexes, numbers, they have a meaning ... I always use the indexes in the 
beginning of [the] school year in our pedagogical meetings. One of the 
first things I do is to show the indexes. Both the IDEB and other general 
indexes and government measures, as well as our internal indexes, reten- 
tion, and dropout indexes, and what we will do in order to keep students 
at our school. 

(BR-SS-1) 


In China, our respondents mentioned various school initiatives such as the cre- 
ation of extracurricular classes which were launched to gain additional scores 
in external evaluations. We witnessed no example of the appropriation of new 
QAE tools for internal school purposes. 


Schools’ room for action in relation to resources 


The connection between evaluation results and resource distribution allows 
some schools and school workers to benefit from the new QAE system. 
Some interviewees said that students’ high performance could secure the 
performance-based element of a salary (in Russia and in some Brazilian 
localities) and the obtaining of higher qualifications (in Russia and China). 
Low performance was perceived by our Russian and Chinese respondents as 
a potential threat to securing necessary funding, which we did not observe in 
Brazil, where low performers tended to receive assistance rather than punish- 
ment. Although Brazilian schools can obtain resources if they perform poorly 
in evaluation schemes, maintaining low scores in national and sub-national 
standardised tests is far from becoming a strategy for such schools to supply 
their needs. In Russia, where examination results carry high stakes for students, 
schools can also attract additional financial resources by offering exam prepa- 
ration classes for a fee. 


The principal very actively encourages additional paid-for classes. In grades 
9 and 11 they are not really voluntary, and in other grades they are also 
strongly recommended to students, because it is very important for the 
school budget. 

(RU-S-26) 


Schools’ high performance also indirectly creates opportunities to obtain better 
resources. First, it helps to attract better teachers, because salaries, career devel- 
opment, and reputation depend on performance evaluation results. It also helps 
to attract performance-oriented students. In Russia, schools receive per capita 
funding, so having more students is financially beneficial. This also applies to 
non-public schools in China (“non-public” includes expensive private schools 
and cheaper non-government schools, such as Minban schools): 


School reputation is very important to us. Even if there were no inspec- 
tion from the government, we would still do our best to improve quality 
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because unlike public schools, our lives ... [depend] on students and how 
many students come here. 


(CN-MBS-01) 


Reputation and influence 


School performance is converted into school reputation through unofficial media 
rankings (in all three countries), public self-evaluation reports (in Russia and 
China), and participation in contests and projects conditioned by performance 
(in all localities). A good reputation is itself valuable, as was frequently emphasised 
by our Chinese interviewees. It also motivates school personnel and students. 
A Russian teacher explained why high performance in examinations was impor- 
tant: “It is our school’s prestige, and parents expect it from us, and my reputation 
in the city — I want to support it, not to lose it” (RU-S-27). Creating a good 
reputation and visibility is a long-term strategy to attract resources and influence: 


You [as a school] should participate in contests, be visible, so that they [the 
authorities] see you, remember you afterwards. There are contests such as 
“Teacher of the Year” — we cannot hope to win ... those. But there are so 
many different contests, you can find one that does not depend on results [of 
students so much],in which you can win, if you read the criteria attentively ... 
Piloting [of federal or regional educational initiatives] — they [the authorities] 
don’t give it to any [random] school. And ifa school participates in piloting, if 
it organises city and regional seminars — the teachers [of this school] can then 
mention it in their qualification documents or in grant applications. 
(RU-S-4) 


Through participation in projects, contests, and so on, school administrations 
develop networks and coalitions which eventually help the school to infor- 
mally influence education decisions and resource distribution. Another infor- 
mal channel for schools’ local and even regional influence is through influential 
parents of students, teachers (especially honoured teachers), or any prominent 
figure’s participation in school activities (for example, war veterans participating 
in patriotic upbringing, sports champions invited to school competitions, or 
local business leaders invited to graduation events). Demonstrating high perfor- 
mance and creating local and regional visibility allows schools to attract power- 
ful people to their informal networks. 


The head of [the] city administration called me and reprimanded me: 
“Why do you solve your problems through veterans?” — but I didn’t! We 
just invited this veteran, and he was in three wars, he is 96 years old, he met 
with Putin, we regularly invited him to school events, so he also wants to 
do something for us. He asks what we need, and we haven’t yet got [good 
sports facilities]. 

(RU-S-1) 
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Our observations in all three localities provided examples of schools with strong 
performance-based reputations which acted as “consultants” or “best practice 
providers” in their locality. In some cases, local education authorities delegated 
power to top-performing schools by making them official providers of exper- 
tise and advice (noted in all three countries). 


The school that receives the bonus [for their high performance, in a scheme 
that is still under discussion] must, as . . . compensation, offer some support, 
as... [a] “sister school” to each school that hasn’t been able to achieve the 
goal, in order to allow them to grow together. 

(BR-S-01) 


To be frank, our [school’s] quality and standards ... have exceeded those 
required by the inspection and evaluation. I think the function [of inspec- 
tors] to “guide and supervise” is [less applicable] to us, unlike weak schools 
that are supervised and guided by education inspection. Schools like ours 
are more of an example and something to be exported. 

(CN-S-01) 


Low-performing schools’ room for action 


The interdependence of financial, human, and symbolic resources leads to a 
situation where school performance evaluation results sustain a virtuous cycle 
of resource accumulation for high-performing schools and a vicious cycle for 
low performers, who are increasingly disadvantaged because of their students’ 
poor results. Teachers in these schools cannot improve their qualifications, as 
they have few if any high-performing students. Hence, it is harder for schools 
to attract new teachers, not only because career opportunities are bleak but 
because work itself can be more demanding and less emotionally rewarding. 
There are also fewer opportunities to win in contests; in some cases, schools 
cannot even apply if their previous performance is poor. 


It can be so that a teacher does not participate in any contests, but 
can explain well, is a good teacher .. . I wish they would abolish those 
[performance-based principles of calculating] salaries, they only provoke 
conflict. Or . . . [define] criteria in a different way, or .. . give it to the 
school, so that we could ourselves evaluate our teachers. 

(RU-S-2) 


Implementation of QAE procedures is regulated, so schools have no option but 
to comply with them. 


As for the documents [regulations] coming from above, some of them 
are very concrete, they are very comprehensive, for example, there are 
over forty items in the students’ quality education regulations for school 
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operations pushed forward by the province level. We cannot violate [any] 
of them, otherwise there will be punishments [for] us. 
(CN-S-06) 


Once every three years we have inspections, you know, [to see] whether we 
work in accordance with the licence, with the accreditation. 

If our institution somehow violates some norms, we may lose our 
accreditation, we may lose our licence. Hence all these monitoring studies 
[monitoringi], self-evaluation reports, all these different reports [exist] — all 
this is just so that the institution works as it should work by law. 

(RU-S-2) 


Complying with regulations and government priorities, as well as improving 
numerically measured performance, carries high stakes even for well-established 
schools. The Brazilian data provided an example of how a supervising authority 
restricted resource access to a medium-performing school because it was pur- 
suing its own principles in managing education (retaining low-performing stu- 
dents and those who did not reach the minimum required school attendance). 


I had to go to the Education Secretariat in the evening ... to take the 
documentation, a process of around two or three hundred pages, to prove 
we did it right ... I was, in some way, harassed during the meeting to 
override teachers’ decisions ... to change the results of the teachers’ 
meeting that retained around thirty students . . . under the allegation that 
this [retention of students] would lower the school index [IDEB] ... 
With an intimidating speech [they said] “If you have any projects going 
on at school and the IDEB is [ongoing], and you are retaining students 
and the IDEB decreases ... we will end ... all these projects, we will 
close your labs. 

(BR-MS-1) 


While this Brazilian school openly resisted new evaluation policy and con- 
fronted the authorities, in China and Russia, we witnessed no example of 
schools refusing to comply with QAE policies or openly questioning them. 
However, evaluations may be resisted in hidden ways; in both Brazil and Rus- 
sia, our respondents mentioned that evaluation results could be fabricated and 
that it was to some extent possible to retain traditional practices while formally 
implementing new regulations. 


I believe it is illusory to think that there is total control through the [exter- 
nal] evaluations. Lots of schools and teachers find ways to manipulate the 
dynamics, including making the evaluation look better than what . . . the 
students answered ...in the evaluation. 

(BR-MS-2) 
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Conclusion 


A comparative study of local practice in Brazil, China, and Russia reveals that 
local governance through QAE is more multifaceted in all three countries than 
is nationally envisioned. The enacted QAE policies and their effects on schools 
do not stay within the limits outlined by policymakers. QAE instruments, such 
as large-scale assessments, and the data they produce are reinterpreted locally 
in accordance with existing practices of quality control and school governance 
and are biased towards the political interests of local actors who seek resources 
and power. The opening or restriction of political opportunities for schools 
therefore also seem to differ from national policies’ original intentions. We 
demonstrate that a combination of governance theories, rather than one theory, 
is needed to understand the diverse ways in which QAE policies can enhance 
and change local governance. We also argue that the study of schools’ political 
room for action constitutes an important dimension in the investigation of local 
policy effects. 

We identify several ways in which local education authorities can use QAE 
for governance. First, by evaluating schools and training them to work with 
data, local authorities seek to change internal school processes, make schools set 
specific goals, and focus on problems they have identified. They also anticipate 
that schools will adopt a new regime of constant self-improvement and an 
evaluation culture and that teachers and administrators will inform themselves 
in new ways (by collecting and processing quantitative data) about students’ 
learning gaps. Such aspirations are representative of governance at a distance, 
which aims to change subjects’ behaviour through constant surveillance and 
their “voluntarily” committing to act in accordance with government designs. 

Local authorities also use evaluation in connection with traditional con- 
trol measures (inspections, restrictions, fines, appointing and dismissing school 
administrations) to justify political decisions. QAE thus reinforces authorities’ 
control over schools. Evaluation also provides the ground for resource distribu- 
tion (financial, material, and human) and access to rewards: performance-based 
funding and salaries, higher qualifications, and prestigious status. The provi- 
sion of incentives to comply with new regulations, while allowing subjects a 
degree of operational freedom, characterises a governance model based on new 
public management. We found that some schools’ local governance QAE prac- 
tices could be viewed as “micro-level NPM”, while QAE mechanisms shifted 
responsibility for quality from local authorities to schools. 

In these ways, QAE policies can enhance traditional governance instruments, 
create new connections between implementation of regulations and provision 
of resources, and facilitate greater access to schools’ internal processes, formalis- 
ing them and subjecting them to authorities’ control. QAE tools can also help 
local authorities to demonstrate their own efficiency in supervising organisa- 
tions, to be accountable to the public, and to change interactions between 
schools, authorities, and the community. 
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Brazilian, Chinese, and Russian schools are implementing QAE policies to 
comply with national and sub-national legislation and as a response to the gov- 
ernance measures we have described. Compliance with QAE policies opens 
exclusive access to various resources and powers to high-performing schools. 
Apart from utilising the “official” schemes of resource accumulation (per capita 
funding, performance-based salaries, improving teacher qualifications), schools 
may find ways to use QAE instruments to attract additional resources from 
students and parents. Improved visibility through participation in contests and 
rankings allows schools to assume new roles as experts and providers of best 
practice and to attract affluent people to their networks and thus increase their 
local influence, which facilitates further accumulation of resources. Reputa- 
tion occupies a central position in our analysis because it functions as both a 
symbolic resource and a source of power. We observe that reputation is key to 
virtuous (in the case of high-performing schools) or vicious (in the case of low- 
performing schools) cycles in which schools find themselves ensnared when 
schemes involving performance-based funding and salaries are implemented. 
In some cases, the schools we analysed also embraced QAE policies as a helpful 
tool for quality improvement and internal management. 

The room for action of the schools which opposed new QAE policies 
appeared limited in our research results. Some schools resisted QAE policies 
because they contradicted the school’s educational goals (for example, the Bra- 
zilian school which insisted on retaining students) and faced the consequences, 
while others made no attempt at resistance even when they disagreed with 
policies, because they were governed by national regulations. The multiple gov- 
ernance tools of local authorities appeared effective in enforcing QAE policies 
in all the schools we observed. However, we also observed that schools could 
practise hidden resistance and to a certain extent avoid QAE tools’ penetration 
of schools’ internal processes. 

In revealing the different mechanisms of QAE policies’ local influence, our 
findings raise questions for further investigation. How typical of these and other 
localities are the effects we observed? What conditions authorities’ and schools’ 
selection of specific mechanisms? How are relations between local education 
actors influenced by broader contexts, for example, by local authorities’ degree 
of autonomy from national and sub-national government and that of schools 
from different levels of government, or by national and local governance lega- 
cies? Who benefits from QAE policies in different situations? Answers to these 
questions and more would greatly enrich the understanding of local education 
governance through QAE. 
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of quality assurance 
and evaluation 
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Introduction 


In this chapter, we analyse the views of Brazilian, Chinese, and Russian actors 
concerning the future development of quality assurance and evaluation. Our 
interest is in examining actors’ perceptions within a wider historical and politi- 
cal context, with the aim of comparing the possible futures of QAE in the three 
case countries. The chapter complements the findings presented in previous 
chapters by focusing not only on past experiences but also on the future aims 
which guide present policies and action (see Adam 2010; Andersson 2012; 
Andersson & Rindzeviciuté 2016). By looking at the future goals set for edu- 
cation QAE, we aim to advance research into one of the book’s key themes, 
namely, how QAE policies shape education practice and discourse. 

We draw on critical futures studies (Bussey, Inayatullah, & Milojevic 2008; 
Ogilvy 2004), which perceive the future as a site of alternatives, instead of 
attempting to predict it. Moreover, we subscribe to the argument that the task 
of the research on possible futures is not only to contour the unknown but to 
rethink the structures of power and governance embedded in official future 
scenarios and forecasts (Cruz 2015; Inayatullah 1990, 2002; Slaughter 2002). 
These ideas are related to the theoretical notions of complexity referred to in 
the book’s earlier chapters, one of their key themes being the future’s unpredict- 
ability (Biesta & Osberg 2010; Cairney & Geyer 2015). Predicting the future of 
complex systems is difficult, which is why researchers who recognise this chal- 
lenge refrain from definite forecasts and focusing on alternative future images 
and scenarios. The idea of alternative futures is also reflected in this chapter’s 
title, which refers to the differing perceptions arising from the research data. 

Investigating the future as a site of alternatives helps in grasping a picture 
of large political changes and identifying patterns in current political action 
affecting the future (Cruz 2015). It also helps in understanding what future per- 
spectives become hegemonic over others and why. The fall of socialist utopias 
at the turn of the 1980s and 1990s, for example, created room for the rise of a 
neoliberal, globalised, and technologised view of the future (Milojevic 2005). 
This neoliberal thinking grew stronger until the recession of the 2000s, after 
which criticism of the social disadvantages caused by neoliberal economic poli- 
cies increased, particularly in Latin American countries (Coronil 2011). 
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We agree with Masini (1993) that views of future and time are strongly linked 
to their surrounding cultures and traditions. Several factors, such as experience, 
and varying philosophical and political traditions and systems, underlie the dif- 
fering perceptions of the future in our case countries. An understanding of 
these differences requires knowledge of the countries’ own traditions, as these 
belong to the collective memory affecting present politics and future develop- 
ment (Hoffman 2013). Moreover, some elements of views of the future may 
have evolved over a long period, and sometimes visions which had little impact 
originally may have become mainstream (ibid.). This finding is related to the 
idea of path dependence and increasing returns, where large consequences may 
result from relatively small or contingent events (Pierson 2000). 

The first chapters of this book comprehensively described our project’s data. 
Here, we need to point out that not all the interviews conducted for the pro- 
ject referred to the future. The analysis for this chapter is based on selected 
national-level interviews in which such references were made: Brazil (n=22); 
China (n=15); and Russia (n=11). With this number of interviews, we of course 
cannot claim to demonstrate “how educationists in general in Brazil, China, and 
Russia think about the future of QAE”, but our data allow us to reveal the quali- 
tative range of their opinions. In addition to the interview data, we examine the 
latest plans for education, and especially QAE, put forward in each case country. 

Our interview data on the possible futures in the three countries may be 
characterised as follows: in Brazil, interviewees openly expressed their personal, 
rather than organisational or official, views, whereas in China and Russia possi- 
ble futures were mentioned relatively rarely by our interviewees, and such views 
were expressed in response to questions prior to the explicit futures question 
at the end of the interview. To some extent, such views have already surfaced 
in the analyses reported in the book’s earlier chapters. As this chapter’s analysis 
shows later, notions of what constitutes “quality of education” and views on 
QAE are closely interlinked: different conceptions of quality have implications 
for how quality can and should be evaluated, and conversely, different tools of 
QAE either prioritise or downplay different notions of quality. 

We continue with a review of the changing perceptions of time and future 
from an historical perspective, including the archetypes of the future of edu- 
cation developed by Inayatullah (2008). Next, we focus on an analysis of the 
case countries by briefly considering the domestic economic, political, and 
ideological factors which may affect actors’ views of the future development 
of education QAE. Thereafter, we use our interview and documentary data to 
analyse how actors in these countries perceive the future trends of QAE in their 
respective contexts. 


Changing concepts of time and different 
archetypes of the future of education 


Barbara Adam (2010) presents a substantial analysis of how human percep- 
tions of the future have evolved. The earliest conceptions emerged and evolved 
within religious mythologies which determined the world’s future direction 
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and the human being's life course. The annual rhythm characteristic of natural 
economies reinforced perceptions about the cyclical nature of time and the 
recurrence of the past in the future (Adam 2010). The development of science 
and technology and the rise of the idea of linear progress represented the fall of 
myth; the idea of progress guided the search for better utopias and challenged 
religious providence. People now believed they could influence the future and 
that they could remould it to suit their needs. The calendar enabled more pre- 
cise planning, which led to an increase in control, social synchronisation, and 
regulation. At the same time, paradoxically, the future became more uncertain 
and contingent, as previously unchallenged traditions of the past ceased to pro- 
vide answers to future problems (Adam 2010; Ogilvy 2004). 

The beliefin constant progress has met increasing criticism in recent decades. 
Ecological devastation, global economic crises, and the failure of socio-political 
utopias has strengthened the post-modern view that the world and its future 
cannot be explained by relying on the idea of progress (Adam 2010; Miloje- 
vic 2005; Ogilvy 2004). Adam (2010) notes that the future should no longer 
be considered a “void”; she finds the idea of a conjecture or an “open future” 
flawed, as present and past generations, through their irresponsible exploitation 
of nature, have already imposed limits for future generations. Ogilvy (2004) 
even describes the present enthusiasm for predicting possible futures — various, 
often conflicting scenarios and future visions — as tragi-comic. 

Current prospects, as Milojevic (2005) states, tend to fall into two categories: a 
firm belief in continued technological development or anxiety about ecological 
regression leading to natural disaster. However, she cautions against succumb- 
ing to such binary utopian or dystopian views. We agree with Milojevic's view 
that instead of taking a binary approach to the future, it should be considered a 
heterotopy, where many different, often competing, visions exist simultaneously. 
These visions are related to values and are continuously negotiated, locally and 
globally. What some mean by utopia may thus mean dystopia to others. 

The theory facilitating this chapter’s analysis is Inayatullah’s (2008) “futures 
triangle” and the five archetypal images of the future and the position of edu- 
cation which accompanies it. The futures triangle maps the future in three 
dimensions: the weight of history, the pushes of the present, and the pulls of 
the future. Examining historical trajectories in futures research is important, 
as many of the existing deep structures set the boundaries of future action 
and change. Likewise, identifying existing trends is important because some of 
these trends continue and have an influence far into the future. The archetypal 
images of the future are those which pull people forward (Inayatullah 2008). 
The archetypes of education futures were inspired by the tradition of research 
into various future archetypes (see Cruz 2015) and developed in a series of 
workshops. Although the archetypes are different, they share a common feature 
in their criticism of the existing education system (Milojevic 2005). 

The first archetype believes in constant progress and evolution. It is charac- 
terised by a modernist and industrial vision that education should provide basic 
skills such as mathematics and training for the needs of the economy and the 
nation-state; it should train consumers, workers, and citizens. 
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The second archetype is the opposite of the first and cautions against the 
collapse of humanity which people will inevitably cause in their continuing 
excessive exploitation of nature. Humanity has overshot its limits; the future will 
be worse than the present. This archetype portends growing inequalities among 
people and deteriorating future conditions. Current education and training pro- 
vide no remedy for social problems and do not meet a changing world’s needs. 

The third archetype is idealistic: the world manifests itself as a garden, “Gaia”, 
where cultures are flowers and people’s inequality can be socio-technologically 
fixed. This requires changes in education and training; education institutions 
should transform themselves from silos of instruction into ecologies of learn- 
ing pedagogies. Education is idealistic; schools and universities become gardens 
where students focus on finding their unique skills and learning from each other. 

The fourth archetype is globalism: when borders fall, free movement of tech- 
nology and capital will bring wealth to all. This archetype is opposed to tradi- 
tional “isms” and dogmas, which it considers as barriers preventing people from 
achieving a new world. Education prepares individuals for the world of global 
technology, in which new technologies, global corporations and businesses, and 
non-governmental actors define frames of reference and rules of action. 

The fifth archetype constitutes a counterbalance to the other archetypes by 
addressing the future nostalgically. A simpler way of life lies ahead, with a clearer 
hierarchy and less disruptive technologies. The best days have passed, and the 
changes of the present are too overwhelming. Education should therefore 
return to its fountainheads, which are filled with basics focusing on morality, 
clear gender roles, strong leadership (often male), and communitarian values. 

In the following section, we apply the futures triangle and the idea of the 
five archetypes to a reflection of our analysis of QAE in Brazil, China, and Rus- 
sia. We are especially interested in whether any of these archetypes manifests 
itself in the country contexts. Based on previous research, we first analyse the 
historical and political factors which have affected images of future education 
and QAE development in the three case countries. We then examine how 
the actors interviewed for this study see the possible future trends of QAE. 
We also consider the possible futures expressed in the latest national plans for 
education. These are the National Education Plan 2014-2024 (PNE) in Brazil, 
the Outline of China’s National Plan for Medium and Long-Term Education 
Reform and Development (2010-2020), and the State Programme for Educa- 
tion Development in Russia (2013-2020). 


Perceptions of the future of QAE in Brazil, 
China, and Russia 


Brazil: education as a national commitment 


Historical weight and pushes of the present 


In Brazil, the contemporary period is felt by many as a transitional stage to 
the future (Coronil 2011). However, this societal sense of “being in transition” 
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is not new. The Brazilian narrative has always depicted the present as a step 
into the future rather than a continuation of the past or a moment of its own. 
A few examples of recent official discourses under democracy clearly illustrate 
this constant reference to an abstract future: in 2011, the legacy of Cardoso 
(1995-2003) was characterised as “sowing the future” (Graziano 2011);in 2013, 
ex-President Lula da Silva (2003-2011) told a group of French entrepreneurs 
“there is no other country with the perspective of the future that Brazil has” 
(Silva 2013); in 2014, ex-President Rousseff (2011-2016) said that “education 
is our passport for the future” (Rousseff 2014); in a recent event called “Pro- 
ject Brazil Future”, President Temer (2016-present) presented some economic 
ideas drawn from a programme called “A Bridge for the Future” (n.a. 2015). 

Although opinions diverge (Braathen & Kasahara 2015), Brazil has always 
characterised itself, and also been characterised, as the land of the future 
(e.g., Zweig 1941). Early Brazilian history is associated with colonialism and 
its later history with agrarian oligarchies and dictatorship, which provided no 
pertinent model for present or future action. The building or fostering of tran- 
sitional stages to the future through a — frequently rhetorical — discursive con- 
struction may be understood as an inherent strategy to discursively break with 
history. Transitional stages channel attention and effort towards the construc- 
tion of a better or promising future. This confidence in the future forms part 
of a set of societal dynamics and socio-cultural features which emerged in, took 
form in, or permeated Brazilian education history. These shaped the current 
views of the future in education, as the next section shows. 

The Brazilian future has been constantly imagined in connection with the 
country’s socio-economic development. Throughout the twentieth century, a 
focus on industrialisation, modernisation, and growth permeated every gov- 
ernment’s political project regardless of its political stance, albeit differently. 
Although the main goal of these development plans was to attain socio- 
economic progress, they always included education concerns, and changes 
progressively occurred in schooling despite the huge gap between political 
intention and action. This close discursive connection between education and 
socio-economic progress allowed education to remain on the political agenda 
in times of both democracy and dictatorship. Despite the political discontinuity, 
the democratisation of education, seen in the concrete expansion of education 
opportunities, continued (Oliveira & Araujo 2005; Klein 2006; Oliveira 2007; 
Gouveia & Souza 2013). 

However, the universalisation of education progressed very slowly. There was 
always significant political interest in education in Brazil, and policy decisions 
were always based on political rather than public policy reasoning. Govern- 
ments tended to favour their own political base (Plank 1990) and undertake 
development plans which reflected these political interests; the place of educa- 
tion in public policy mirrored the political power of the time, which influ- 
enced the allocation of public resources to education development (Wjuniski 
2013). Education has developed according to governments’ ideological stances 
and political interests (Fonseca 2009). Although the level of political discon- 
tinuity was not drastic in the education policies of the last three democratic 
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governments (1995-2016), civil society and scholars alike pointed to discon- 
tinuity at the level of concrete programmes as a major problem in Brazilian 
education (e.g., Zarpellon 2011). 

Besides governments’ own political interests, there have always been ideo- 
logical and political differences across sectors, political levels, and the country. In 
the past, although clearly favouring specific societal groups, governments sought 
certain political compromises (Saviani 1996). Recently, governments have con- 
tinued this path of seeking compromise through the fostering of democratic 
forums in which education plans and programmes are discussed. The most 
important are the National Conferences of Education (CONAE), which gather 
more than 3,600 participants. The current National Education Plan 2014-2024 
(PNE) was outlined im the 2010 CONAE, ratified by the 2014 CONAE, and 
thereafter legislated. The PNE is thus the result of a broader democratic com- 
mitment, which explains its broad acceptance across the education arena. 

As Chapter 3 explained in detail, the PNE endorses the use of QAE to 
achieve quality in education (Brasil 2014; Brasil, MEC/SASE 2014). This 
largely entails large-scale assessments, as students’ performance is considered 
an important indicator in quality evaluation. The Development Index of Basic 
Education (IDEB) is perceived as the main Brazilian education quality indica- 
tor, and the PNE has restated the IDEB’s target of meeting the Organisation 
for Economic Cooperation and Development’s (OECD) Programme for Inter- 
national Student Assessment (PISA) mean score by 2021. Despite the primary 
focus on indicators, the PNE also envisages other QAE practices. The most 
significant is anew method of calculating the education budget which regulates 
the amount spent per student and per year for assuring the minimum standard 
of quality education (see Centeno, Kauko, & Candido 2017). This alternative 
method requires a substantial increase in education funding. 


Perceptions of the future 


Inayatullah’s archetypes (2008) help in defining Brazilian views by opposition; 
it explains views which do not seem to belong to the Brazilian culture. This 
section shows that “collapse” no less than “back to the future” seems far from 
adequate in defining the Brazilian approach to manifesting or perceiving the 
future. Neither disenchantment with education nor nostalgia plays a role in the 
views expressed by Brazilian interviewees. On the contrary, education is now, 
as always, an important socio-political project whose aims are both individual 
fulfilment and the country’s progress. 

Indeed, interviewees were quite hopeful about the future while acknowl- 
edging difficulties and limitations. One interviewee aptly remarked that the 
Brazilian narrative had always been one of improvement (BR-NNGO-04). 
Most interviewees mentioned the ongoing democratisation of education in 
Brazil — the progressive focus on school access, dropout and retention rates, 
and recently on school quality — to justify their confidence in continued future 
improvement. 
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Where recent developments were concerned, they mentioned two favour- 
able factors as crucial for the improvement of education quality. These were the 
auspicious socio-historical moment and the appropriateness of the PNE. The 
interviewees considered that the early 2010s represented an auspicious moment 
for the tackling of education reform. The huge attention paid to education 
and the concomitant engagement of every sector and level of society in the 
education debate provided an excellent basis for improving education policy 
and practice. One interviewee concisely summarised this in saying that in the 
Brazilian imagination, education would save the country (BR-N-09). 

All interviewees viewed the PNE very positively, and all considered it the 
product of unprecedented collective understanding and commitment. They 
approved of the PNE’s goals and targets and maintained it provided the neces- 
sary framework for Brazilian education. Regardless of their room for action or 
professional profile, all interviewees declared that Brazil had now clearly and 
correctly defined its education problems and solutions and the paths and direc- 
tions which should be followed. 

Interviewees expressed their views of the improvement of education quality 
in general; they did not focus only on QAE improvements. QAE was mostly 
linked with large-scale assessments, which in turn were connected with learn- 
ing attainments. In their eyes, national evaluation, and the consequent indica- 
tors and indexes, belonged to a larger framework, whose main goals were the 
improvement of education quality and the enhancement of pupils’ learning. 
The intensity or frequency of the relationships articulated between QAE and 
education quality varied — as might be expected — according to the interview- 
ees’ room for action: the more individuals were involved in QAE practices, the 
more their narrative swung towards the significance of this relationship. Thus, 
while those in charge of large-scale assessments tended to emphasise QAE, 
those with no direct connection with QAE practices tended to make only brief 
narrative incursions into the topic. 

What may be unexpected is that, despite criticising the Evaluation System of 
Basic Education (SAEB) and related indicators and acknowledging its limita- 
tions in assessing education quality, all interviewees considered the existing QAE 
instruments to provide an adequate picture for monitoring quality and equality 
in Brazilian education. Many considered that it also provided valuable informa- 
tion to guide education practice and education quality improvement policy. 

However, QAE practices were mentioned less than other factors connected 
with education and school quality. Interviewees addressed concrete education 
developments and tended to elaborate their views based on concrete examples, 
while addressing broader developments, such as the universalisation of school 
access, the reduction of inequality, and the need for change in school and teach- 
ing culture. In referring to more concrete goals perceived as more attainable, 
interviewees linked their views to learning achievement and education equality 
across the country, democratically consolidated and sustainable political pro- 
cesses, appropriate education policy and practice, school conditions, teacher 
training, working conditions, and teaching quality. 
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In brief, QAE practices were only one of many factors interviewees men- 
tioned. Many answered in a quite utopian tone, referring to major or encom- 
passing abstract socio-political and even cultural changes. These references 
simultaneously conveyed their utopian and dystopian views. One interviewee 
clearly expressed this by mentioning that numbers helped to identify the “non- 
learning” and the dropouts, to whom Brazilians should show no tolerance 
(BR-N-03). However, this interviewee concluded that this was a dream, cor- 
responding to an education system in which everyone from the school to the 
federal government committedly fulfilled their own role. 

Interviewees’ dystopian views were largely linked to a lack of political con- 
ditions and real collective agreement on the implementation of practical meas- 
ures which addressed quality and equality in such an enormous and diverse 
country. The political discontinuity and divergent positions and views which 
have converged in the PNE’s formulation continue to diverge in its execution. 
Another common feature mentioned was the slow pace at which changes 
have occurred and reality has been impacted. Interviewees mentioned that 
improvement would continue to occur as it had in the past, despite economic 
and political cycles. However, they foresaw a return to a very slow pace of 
change. 


China: education as a means of rejuvenating the country 


Historical weight and pushes of the present 


In China, future visions bear complex semantics inherited from the traditions 
of philosophical thinking in ancient and imperial times. Hoffman (2013) pro- 
vides an interesting analysis of how images of the future have changed in China 
and the effects these changes have had on today’s discourses. She points out that 
different concepts of time have been intertwined and manifested in various 
ways throughout Chinese history. 

Confucian philosophy, which originated around 500 BCE, was founded on 
a linear concept of time and believed that the future was secured when people, 
led by rulers and officials, lived up to the highest ethical standards. Confucian 
teachings proclaimed that the golden age had passed and the imperial rulers 
should therefore always look to the past in reforming the state. This eventu- 
ally led to the end of the imperial era, because it provided no tools for the 
state’s regeneration. The new republican government sought influences from 
the West, providing an impetus to a discourse on the importance of external 
influences which has continued to this day (Hoffman 2013). 

The Chinese Communist Party’s assumption of power in 1949 represented 
a profound change in China’s history and its conception of the future. The 
dismantling of archaic traditions and the later inception of reform became pri- 
orities. Mao Zedong’s utopian projects, such as the Great Leap Forward and the 
Cultural Revolution, underlined this approach (Hoffman 2013). Since the late 
1970s, China has pursued modernisation without abandoning the one-party 
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system. Modernisation, prosperity, the reduction of regional disparities, and the 
safeguarding of the country's social cohesion have guided Chinese thinking 
about the future in the last three decades. 

China is considered to be standing at a crossroads. The aspiration to influence 
future development through five-year plans, legislation, and long-term outlines 
guide societal policies and decision-making (Yang 2014). Futures research is 
popular and is closely linked to political decision-making and debates. The 
public approach to the future is generally optimistic, arising from the one- 
party system (Callahan 2016). However, the population is ageing, the country's 
economic growth has slowed, and growth in military spending is the world's 
largest. The highly educated urban middle class faces a crisis caused by rising 
prices, and the countryside cannot provide meaningful work. People are weary 
of the idea of “growth first” (Hoffman 2013). 

Current official thinking about the future is crystallised by the Communist 
Party's slogan the “China Dream”. This official future vision was originally pre- 
sented in a book by Colonel Liu Mingfu, and President Xi Jinping subsequently 
used it in his inauguration speech. X1’s China Dream seeks the return of China 
to its natural position as one of the world’s leading states, a reference to the 
country’s imperial status in the nineteenth century, when China accounted for 
about one-third of the entire global market. The Chinese Dream implicitly sees 
China superseding the USA as the leading economy by 2035. The aim is also to 
build a moderately prosperous society by the Communist Party’s centenary in 
2021. Concrete goals are the doubling of gross domestic product, 60% urbani- 
sation, the building of a space station, and the blueprint for the One Belt One 
Road Initiative. The Chinese Dream differs significantly from previous harmo- 
nious visions of the Chinese future: it is seen more Sinocentric in its endeavour 
to consolidate China’s position internationally (Callahan 2016). Confucianism 
has enjoyed a robust revival in this wave of nostalgia (Deng 2011). 

Hoffman (2013) notes that the China Dream is a vision not of individuals 
but of the state. She describes how, alongside the formal vision of the future, a 
vision of the “Chinese Dream” exists which centres on constitutionality, free- 
dom, and democracy. However, although some academic circles and public 
blogs discuss this vision, it is the view of few Chinese. Moreover, many different 
influences from ancient and recent history are also present in current Chinese 
thinking: in recent decades millions of Chinese have turned to spiritual tradi- 
tions to navigate the future (Hoffman 2013). 


Perceptions of the future 


Education, together with science and technology, has become the key means 
for the restoration of China’s leading global position (Postiglione 2015). The 
idea of rejuvenating the country through science and education was inherited 
from the Deng Xiaoping era of the 1970s and 1980s and strengthened dur- 
ing Jiang Zemin’s presidency in the 1990s and early 2000s (Yang 2014). The 
idea that education is a means to rejuvenate the nation is also incorporated in 
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the Preamble of the Outline of China's National Plan for Medium and Long- 
term Education Reform and Development (2010-2020) (hereafter “Outline”; 
Gu 2010). This document covers every level of education from pre-school to 
higher education and was created in cooperation with the OECD, the World 
Bank, and other international actors (Gu 2010). Shanghai's participation in 
PISA 2006 also provided information for the Outline’s preparation (Baird et al. 
2016). The document contains several recommendations intended to enhance 
the quality assurance and evaluation of education. 

The Outline includes several strategic goals, two of which are the equal- 
ity and quality of education. These resemble the main themes of PISA (see 
Schleicher & Zoido 2016; Gu 2010). The Outline states that the quality of 
education should be improved through several measures, such as the reduction 
of students” workload and the establishment of state standards for urban and, 
whenever possible, rural schools (Gu 2010; Outline 2010-2020). Subsequent 
studies have highlighted how the Outline strives for policies which reduce gaps 
between urban and rural education provision and encourage a transformation 
from exam-oriented to quality-oriented education (e.g. Yuan 2013). The Out- 
line has boosted curriculum reform, which is a concrete tool for improving the 
quality of education (Xin & Kang 2012). 

A closer examination of the Outline reveals a contrast between quality edu- 
cation and the measures which guarantee it. The document paints a picture 
of an expanding monitoring and evaluation apparatus. It stipulates that teach- 
ers’ qualifications will be improved, teacher recruitment will follow rigorous 
practices, and the development of evaluation will be continued by setting sci- 
entific and diverse benchmarks. Diverse evaluation approaches which help to 
promote students” development will be explored, student records kept, and the 
assessment of comprehensive quality developed. Attention is paid to the quality 
of the supervision system. The Outline stresses that mechanisms for effective 
supervision at all levels will be established and that schools should “willingly 
accept and cooperate with legislatures at all levels in supervision and inspection 
of education law enforcement, and with the personnel sent by law-enforcement 
agencies on supervision visits” (p. 43). 

Actors interviewed for this study were moderately critical but also optimistic 
concerning future education reform. Some interviewees noted that the lack of 
independent evaluation authorities and supervisors’ lack of expertise slowed 
the development of quality education. The improvement of supervision was 
important because supervisors should “possess a capacity to put forward plans 
to improve schools in the future” (CN-E-07). Some interviewees were criti- 
cal of education reforms as generally insufficient and focused only on minor 
structural changes. They anticipated a deepening of urban and rural inequality 
if thorough measures were not undertaken. Old traditions such as the influence 
of students’ socio-economic background on school choice were deeply rooted 
in the education system. The greatest challenge lay in the reform of antiquated 
approaches to learning. 
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For the optimists, China's economic and technological development provides 
the necessary resources to ensure the implementation of evidence-based educa- 
tion and training reforms. Interviewees highlighted the importance of recruit- 
ing qualified teachers, the development of teacher training, and the move from 
exam-oriented education to an emphasis on the child's overall development. 
Some interviewees placed much trust in the National Assessment of Education 
Quality (NAEQ); this assessment was believed to enhance the monitoring of 
not only the teaching and learning process but also results — essentially a control 
of quality. In this respect, the NAEQ resembles a (social) technology which is 
believed to solve quality problems in remote areas as well (see Baird et al. 2016). 

Although interviewees’ opinions on the need for reform seemed harmoni- 
ous, they differed concerning the pace of measures. Some pointed out that swift 
political approaches were needed if the disparities between rural and urban 
areas were not to continue to deepen. Schools are already undergoing “a quiet 
revolution” (CN-E-07), in which ongoing curriculum reforms aim to reduce 
students’ workload, especially in repetitive memorisation or drilling, and to 
enhance students’ self-regulation and creativity in learning, as PISA encourages 
(Blackspear 2012; Baird et al. 2016). Other interviewees felt such a fast pace of 
reform was unrealistic. There were, as one interviewee noted, several factors 
which might slow the implementation of future QAE reforms, such as chal- 
lenges related to the decentralisation of the education administration and its 
separation from the supervision system (CN-E-03). 


Russia: appreciation of the national pedagogical tradition 
and ideas for new development 


Historical weight and pushes of the present 


The historical milestones of Russian futures research are found in the Bolshevik 
Revolution of 1917 and the dissolution of the Soviet Union in 1991. The Bol- 
sheviks brought a strong future orientation to the Soviet Union’s social policy. 
At its core was Soviet Communism’s teleological eschatology, in which the idea 
of a utopia built by workers led all political activity. Future studies focused on 
making predictions and assumed an increasingly ideological hue during the 
Cold War (Bacon 2012). 

The collapse of the Soviet Union brought the breakdown of the founda- 
tions of Soviet futures research. Although predictions of the Soviet Union’s 
demise abounded during the Cold War, future studies within and outside the 
country failed to accurately predict the state’s collapse. Bacon (2012) describes 
how futures research worldwide reached its nadir, failing to anticipate the dis- 
solution of the Soviet Union and subsequently of the East European Bloc 
of socialist countries. In the years which followed, linear predictions were 
abandoned and interest in multiple scenarios increased. In the last two dec- 
ades, more than ten divergent scenarios of the future of Russia have been 
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constructed, reflecting the political issues which emerged at the time of their 
drafting (Bacon 2012). 

Historical traditions and political changes have contributed significantly 
to how the future is understood in Russian daily life. According to Schil- 
ling (2008), many factors have influenced how Russians perceive their future. 
These include administrative arbitrariness, a high degree of power distance, 
late industrialisation, and wide geographical extension. A governance culture 
in which strategic decisions can be made in a tsarist fashion beyond legislative 
influence has shaped public views in a direction which makes people feel inca- 
pable of influencing their futures (Schilling 2008). Despite late industrialisation, 
the socialist economy was based on the production of detailed and centralised 
five-year plans, which were usually over-ambitious. Based on anthropological 
research, Yurchak (1997) has called the prevailing attitude towards the state’s 
policies “non-involvement and simulated support”. 

In Russia, official views and presumably also popular thinking about the 
future have significantly changed since the collapse of the Soviet Union. The 
socialist utopia has been replaced by various scenarios, the contents of which 
vary significantly (Bacon 2012). A key feature of Russia's current domestic pol- 
icy has been its division into conservative or Westernised positions, depending 
on attitudes towards foreign models and external influences (Johnson 2010). 
This division has also had implications for perceptions of the future and their 
interpretation. For example, Russia’s political initiatives in recent years, such 
as proposals for closer cooperation with neighbouring countries, have been 
interpreted as nostalgic and conservative efforts to restore Russia as the world 
power it was during the Soviet period. However, as Krickovic (2014) reminds 
us, analyses of Russian policy should not succumb to overly narrow explana- 
tions, because there are many other political factors apart from nostalgia behind 
recent political initiatives in Russia, such as the unstable international situation, 
which impels Russia to seek new partners. 

Current domestic policy embodies a deep-seated dualism which is signifi- 
cant for Russia’s future development. Views are divided between those who 
advocate sympathy for Western values, European reform, and globalisation and 
those who see the increasing internal pressure for reform arising from globali- 
sation and Europeanisation as a threat to Russian sovereignty (Johnson 2010; 
Pavlova 2010; Morgan & Kliucharev 2012) — a similar division was already a 
key element in political and cultural debate in pre-Soviet Russia. In Chap- 
ter 4, we noted that the influence of international organisations on Russian 
education policy was relatively limited in the early post-Soviet years and has 
more recently become even less significant. At the same time, communication 
between Russian and foreign QAE experts has grown. 

Tensions in political decision-making have also affected education policy and 
its vision for the future. Pavlova (2010) notes that the strategic goals to improve 
the quality of education and the idea of education as the key to socio-economic 
development outlined at the beginning of the millennium have never been 
questioned. However, the dualism mentioned earlier has resulted in a situation 
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where goals created at the beginning of the millennium for the development 
of education and strategies based on them have become a hybrid including 
both traditional and modern goals, the latter influenced by Western values (see 
Boguslavskii & Lelchitskii 2016). As the next section demonstrates, such dual- 
ism is reflected in views of the future of QAE. 


Perceptions of the future 


The State Programme for Education Development, 2013-2020, is a 700-page 
document covering all aspects of federally regulated education policy. The pro- 
gramme's overall aim is “to ensure that the quality of education meets people's 
changing requirements and the development objectives of Russian society and 
economy”. The document outlines a comprehensive and uniform system of 
QAE based on the “principles of openness, impartiality, transparency, and public 
and professional involvement”. This system will comprise the existing national 
examinations after grades 9 (the State Final Attestation, SFA) and 11 (the Uni- 
fied State Exam, USE), and the procedures of independent quality evaluation. 
The information provided by these national tools will be complemented by 
Russia's continued participation in international assessments (PISA, Trends in 
International Mathematics and Science Study (TIMSS), and Progress in Inter- 
national Reading Literacy Study (PIRLS) (Government of Russia 2012:p. 218). 

In general, in our comprehensive interview data from Russia, national-level 
interviewees were more outspoken on the issue of the desired futures of QAE 
in education, and views on this topic diverged more than they did among 
interviewees at the local level (the latter are not analysed in this chapter). Many 
respondents personally indicated that they valued the Soviet and pre-Soviet 
pedagogical tradition or referred to extensive media discussion of this topic. In 
this tradition, important elements of QAE are the strong professional autonomy 
of teachers in the assessment of their students and an emphasis on “upbringing” 
(vospitanie), which refers to learning’s non-cognitive domains. These include 
moral or “patriotic” education (see Rapoport 2009), which is not measured by 
international tests or national examinations. 

The experts interviewed presented several QAE issues and pondered the 
solutions leading to new ideas for development. These ideas were not neces- 
sarily antithetical to “nostalgic” views — the former might also complement the 
latter. The notion of education quality embodied in international assessments, 
especially PISA, was seen as providing a scientific basis for improvement of 
quality through testing students’ problem-solving skills. As a counterbalance, 
the importance of Russian cultural heritage, which values literature and read- 
ing for non-instrumental purposes, was also emphasised (see also Piattoeva & 
Gurova 2018). It is noteworthy that the 1998 OECD report on Russia strived 
for a balance between acknowledging positive aspects of the Russian pedagogi- 
cal heritage, including assessment practices, and criticising the same pedagogy as 
unsuited to the needs of a market economy and international competitiveness 


(OECD 1998: pp. 9, 16, 91, 99-103). 
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Some experts argued that assessment should be more independent of teach- 
ers, because it was the results of their work that was being assessed. At the same 
time, there will be a need to familiarise teachers with the new methods of 
QAE and motivate them to use their results as feedback to stimulate their own 
professional work. Furthermore, the high-stakes nature of the national exami- 
nations at grades 9 and 11 was criticised as narrowing the concept of quality 
to what the SFA and USE measured. Instead of focusing on only one type of 
uniform measurement of learning, there should be a diversified system of QAE 
which comprised both quantitative and qualitative components (such as essays 
and portfolios), covering a wide range of learning domains in the curriculum 
and extracurricular activities. The experts who voiced an opinion on the rank- 
ing of schools based on QAE results and the resulting competitive atmosphere 
did not question it as such but called for more analysis of differences in learning 
achievement between schools and regions which would make it possible to 
devise corrective measures to support low-achievers. 


Conclusions 


Our analysis has shown that history sets boundaries but also provides sources of 
political legitimation to the three countries’ future imaginaries. Based on the 
data from Brazil, it can be stated that there is no symbolic period in the history 
of the country in which interviewees or the examined documentary mate- 
rial locate the future. In contrast, the Chinese authorities, in justifying future 
projects, appear to rely on the nostalgic idea of restoring the status the country 
has lost. The data on Russia also suggest that the recent past has coloured ideas 
of the strengths of Soviet and pre-Soviet pedagogy. However, the nostalgic 
Chinese and Russian views differ substantially in their semantics from the fifth 
archetype described at the beginning of this chapter; instead of looking back 
to traditional or simpler forms of education, the value placed on national tradi- 
tions in China and Russia is combined with the ambitious aim of improving 
the quality of the national education systems. 

In examining the current factors affecting the images of future developments 
of education and QAE in the three countries, we found that education has 
been regarded as a central tool for socio-economic development, especially in 
Brazil, regardless of transfers of power. In Russia, the need for continuing edu- 
cation reforms is generally acknowledged, although views on how such reforms 
are implemented vary. In China, rapid economic growth and an understand- 
ing of education as its key element have contributed to the universalisation of 
basic education and projects to improve the quality of education. However, 
circumstances and trends such as political discontinuity in Brazil, administrative 
arbitrariness in Russia, or the rigidity of governance and legislation in China 
are features which continue to complicate education’s future directions. 

Along with history and present circumstances, the future is influenced by 
many often concurrent and rival views, which are often presented in the forms 
of plans, scenarios, or visions. These views could be conceived of as factors 
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which “pull” human beings towards particular futures. The archetypes pre- 
sented at the beginning of this chapter attempt to categorise some initial views 
of education futures, which, instead of a strict theoretical framework, serve as a 
reflective basis for this chapter's analysis (Inayatullah 2008). Future research on 
QAE in education is especially challenging from this perspective, because the 
core of the education system is people’s ability to acquire new knowledge and 
on this basis to influence future trends and the emergence of new societal ideas. 

In searching our data, we found that Brazilian views reflected a combination 
of several archetypes. While “evolution and progress” is the main — implicit 
and explicit — stance in Brazilian political and social culture (the Brazilian flag’s 
motto is “order and progress”), its justification is most frequently tied to ide- 
alistic education and social ideals (“Gaia”). If we understand “globalism” as 
conveyed by international QAE practices and policies, and as the expected and 
desired growing participation of civic society in education debates and deci- 
sions, it might be said that in addition to the intersection of the first two arche- 
types — which largely define the Brazilian views — globalism is also gradually 
shaping the Brazilian imaginary. 

In China, as previous research suggests, visions expressed at the state level may 
differ significantly from individual views of the future. The experts interviewed 
for this study shared their views on the need to reform school curricula and 
develop schools as modern institutions, the purpose of which was to promote 
the overall development of children and adolescents. Continuous development 
and progress is also the thread of the current national Outline. However, the 
idea of education as a source of national rejuvenation reflects a nostalgia stated 
in the Outline but which was not communicated by the Chinese interview- 
ees. Although it is impossible to exhaustively demonstrate any competing or 
deviating images of the future based on a small number of interviews, our 
analysis suggests that the views of the interviewed experts on the current state 
and future of QAE in education, especially on the supervisory system, differs 
from the stipulations of the Outline. Furthermore, the globalist archetype of 
the future seems unfamiliar in the Chinese context, although China is actively 
seeking to identify best practices abroad and has lowered the threshold for 
cooperation with international actors. Nor was the idea of schools as gardens 
of learning reflected in responses in the Chinese documents and interviews 
analysed in this study. However, this does not mean that the idea of learning as 
a garden is not present elsewhere in Chinese education thinking; the metaphor 
of the teacher as a gardener dedicated to his or her work is common in China. 

The theme arising from our Russian data centred on two opposing, or as this 
research suggests, somewhat complementary forces: Western assessment sys- 
tems measuring quality as cognitive competences and the survival of the peda- 
gogical and moral education developed in pre-Soviet and Soviet times. The 
Russian interviewees reflected on how the education system might be devel- 
oped in a balanced way, with traditional values and new methods for evaluating 
learning achievements complementing each other. They emphasised that the 
development of education should not focus only on one-sided instrumental or 
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cognitive learning and the evaluation of learning outcomes but also on the cul- 
tural and moral upbringing of children and youth. Thus, nostalgia in Russian 
education does not seek a return to previous systems; rather, it seeks to retain 
some elements considered important in national curricula, teaching, and QAE 
practices. In emphasising the importance of educating civilised individuals and 
the significance of non-cognitive learning outcomes, the ideals of the Russian 
education system seem closer to those of the Brazilian system than those of the 
Chinese system. 

Altogether, our study showed that even if only some of the archetypes or 
their variants can be found in the research data, examining them may help to 
distinguish new nuances, issues, and views which may affect education’s future 
policies and governance. 
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Jaakko Kauko, Risto Rinne, and Tuomas Takala 


Introduction 


The preceding chapters analyse the politics of quality in education in Bra- 
zil, China, and Russia. It has become clear that our premise that quality is 
the most important framing factor in education is correct. In initiating this 
research project, we acknowledged the claims previous studies have made 
about this issue. The shared mind-sets of international organisations’ person- 
nel and schoolteachers, who are almost obsessed with quality assurance and 
evaluation (QAE) whether they view it positively or negatively, confirm the 
earlier hypothesis. We take this a step further, arguing that it is more accurate 
to suggest that it is QAE itself, rather than quality, which has become the cen- 
tral framing factor of education policy. 

In Chapter 3 we took issue with the historical development of this change. 
Although different paths to the use of QAE in education are taken, all involve 
cooperation with international organisations and find it possible to use QAE 
as an ostensible solution to various problems. In addition to the international 
organisations’ influences, as we pointed out in Chapter 4, to varying degrees, 
there are also transnational influences via expert networks. Although these 
three countries have built up their QAE capacity, they have created more room 
for experts, who are more transnationally oriented in their work. In Chapter 5, 
we discussed the potential tensions between experts and politicians arising from 
their different operational logics. 

Our research touched on QAE procedures and their concrete work in the 
socio-political and historical context. As Chapter 2 discussed, in attempting to 
conduct our research systematically, we glimpsed the everyday reality around 
the questions of quality in education. As the empirical work progressed, we 
were able to identify how all the flows of QAE data were described as involving 
friction at every stage: production, availability, and use (Chapter 6). Moreover, 
the closer we got to the local level in Chapter 7, the more complex it became 
in relation to the visions of the transnational QAE agenda Chapter 1 described. 
Although the frictions of data produced something new (Chapter 6), it became 
clear that the schools which opposed change found themselves in the most dif- 
ficult position (Chapter 7). To come full circle, interestingly, future aspirations 
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of QAE in education are strongly linked to the past (Chapter 8), which begs the 
question of what this study can teach us. 

The epistemological premise of this book is the idea of complexity and 
contingency. During the many analysis tracks described in Chapter 2, our aim 
was always to try to understand and comprehensively compare the dynamics of 
the politics of quality in education in Brazil, China, and Russia. Indeed, there 
is an internal paradox in our analysis, akin to that of comparative education as 
a field of enquiry. On one hand, throughout our analysis, we can observe the 
totality of the QAE agenda, and, if we think sufficiently abstractly, we can see 
and construct similarities in it across different contexts. On the other, we wit- 
ness a range and plurality of solutions and use of room for action at different 
levels, especially at the local. Understanding this comparative paradox has been 
the goal of our analysis of these dynamics. 

In Chapter 1, we questioned the idea of transfer. Our comparative paradox 
sheds light on identifying the inherent problems of the idea of transfer, even 
asking, “Should we look for differences or similarities?” We can say there is 
both a clear policy transfer in the transnational QAE agenda and that there is 
not. They exist at the same time, like Schrédinger’s cat, in a box with a poten- 
tially deadly trap, and the outcome is left to probability. However, if we think 
of causality more in the sense of complexity theories, the conclusion that both 
ideas co-exist becomes more reasonable. As we argued in Chapter 1, it can be 
deduced that causality is always a question of probability. The existence of the 
transnational QAE agenda increases the probability that QAE governance tools 
will be implemented, yet local conditions create probabilities (path dependen- 
cies) which steer events in different directions. This chaotic arrangement cre- 
ates the conditions for various dynamics. 


Three dynamics 


We argue that three dynamics can be discerned in Brazil, China, and Russia: 


e Quality assurance and evaluation (QAE) rather than quality itself has 
become the objective of education policy. While quality of education 
remains undefined and contested, QAE becomes the concrete, defined, 
must-do in education and remains uncontested. This is constituted in a 
dynamic of “shared and self-reinforcing goal-setting”. 

e The QAE toolbox does not produce quality as such but rather works as a 
means of controlling the provision of education. From the state’s geopo- 
litical role in the global field to the local understandings of governance, 
QAE is an attempt to tease out desired aspects from the education system 
which are not always connected with education. Regardless of this power- 
wielding aspect, implementation itself is multifaceted and frequently trans- 
national. We call this dynamic “authorising but diverted governance”. 

e In the third dynamic, we find that QAE both destabilises and reorganises 
actor roles. What is interesting here is that state actors, for example, can 
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easily use quality to establish their position, but then find that opening the 
QAE toolbox can lead to the destabilisation of the status quo in the newly 
available space for politicking. We call this “destabilising and reorganised 
role-setting”. 


Table 9.1 summarises these dynamics, which are discussed in more detail in 
the following sub-sections, with some references to our research published else- 
where. Each dynamic reflects the dimensions of CADEP (Chapters 1 and 2) and 
at the same time attempts to illustrate the moving nature of politics. The third 
dimension of politicking is the defining term for the other two dimensions, 
as it describes the movement which occurs in the room for action (self-rein- 
forcing, authorising, destabilising) and is symmetrically (“and”) or asymmetri- 
cally (“but”) related to the other attributes. The political situation describes the 
polity’s construction (shared, diverted, reorganised). The political possibilities 
describe what the target of action is or what is considered the addressed prob- 
lem or solution (goal-setting, governance, role-setting) (see Simola et al. 2017). 
We examine each of these dynamics in the following sub-sections. 


Self-reinforcing and shared goal-setting: QAE not quality 


We have already indicated that QAE is on the agenda of international organi- 
sations, usually emphasising the connection between education and economic 
growth, and the need for QAE procedures in education to make successful pol- 
icy to reach their education aims (e.g., Chapter 1). In Chapter 1, we questioned 
this claim, especially the potential for clear top-down rational decision-making 
based on data. Regardless of this critique, we have found that QAE functions 
as a governance tool which creates a self-reinforcing dynamic, in which the 
need for quality and the need for QAE converge and problems and solutions 
are mingled. 

The QAE agenda is shared and embedded in webs of actors. In Chapter 4, 
we argued that the role of international organisations in policy discussions and 
decisions is clearly variable between our case countries and across time. Con- 
trary to a straightforward policy-transfer notion, our findings show that simi- 
larities between local authorities’ and international organisations’ advice and 
the problematisation of QAE and the design of QAE tools at national levels 
do not necessarily prove a cause-and-effect relationship. Any conclusions on 


Table 9.1 Dynamics in the politics of quality in Brazil, China, and Russia 


Politicking: How is the The political situation: What is structurally Political possibilities: What 


room for action used? possible for the constellation of actors? is considered possible? 
Self-reinforcing and shared goal-setting 
Authorising but diverted governance 


Destabilising and reorganised role-setting 
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this question are complicated by the differing views of key actors on “who 
influenced whom and by how much”. However, it is still correct and relevant 
to speak of a transnational QAE agenda as a network, embodying complex and 
changing relationships between various actors. 

When applied to the general education politics of the three countries, the 
picture becomes more nuanced. Based on document analysis of the Organisa- 
tion for Economic Cooperation and Development (OECD) and the World 
Bank, we have established that these organisations give Brazil, China, and Rus- 
sia the same advice, suggesting they take QAE as a guide for policymaking, 
with the occasional addition that teachers’ incentives should be connected to 
performance (Takala et al. 2018). This agenda has been adopted relatively well 
in all three countries but always with national emphases. Although it yearns for 
some elements of Soviet pedagogy (Chapter 8), the Russian national education 
agenda has changed its justification for QAE from the immediately post-Soviet 
aims of democratisation and school autonomy to the securing of national 
economic growth and global competitiveness (Gurova, Piattoeva, & Takala 
2015). The Chinese quality education reform has followed the transnational 
agenda, based on the idea that testing and evaluation are key to the achieve- 
ment of quality, and international large-scale assessments are viewed positively 
(Suominen et al. 2017; Chapter 4). The Brazilian political and academic elites 
have been connected to international trends in times of political and economic 
stability and instability; QAE policies have mirrored global tendencies (Kauko 
et al. 2016). However, Brazilian QAE discourse is coloured by social terminol- 
ogy such as “social quality” (Centeno, Kauko, & Candido 2017). Interestingly, in 
Chapter 8, we have been able to indicate that these aspirations are also projected 
into the future, tightly connected with QAE, which promises longevity for the 
shared agenda. We can at least conclude that QAE has proven an attractive 
governance tool. 

The agenda feeds a self-reinforcing dynamic. As we hinted at this chapter’s 
start, our research supports the argument that the technical process of measure- 
ment rather than the quality of education itself has become one of the main 
aims of education policy. In Chapter 6, we demonstrated that QAE data fail 
to give the kind of directions to policy or pedagogy which their producers 
claim. The use of data itself becomes a game, based on the “dynamics of trust 
and distrust”. This is also seen at the local level, where there is little possibil- 
ity to avoid QAE but where little use is seen in data collection (Chapter 7). 
The question of the nature of quality in education becomes side-lined in the 
process of its implementation. The results of this project, reported elsewhere, 
indicate that standardised testing feeds a need for more testing (Piattoeva & 
Saari 2018) and that quality becomes simultaneously a means of problematising 
education and providing a solution for it (Minina et al. 2018). In Chapter 7, we 
also pointed out that the schools which opposed QAE reforms faced the most 
difficult challenges. 

Based on our work, we can say that QAE is a global phenomenon and a 
shared goal, and we have observed that it penetrates practice at global, national, 
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sub-national, and local levels. It should be remembered that this is not to say 
anything, yet, of the multiple direct and indirect consequences of QAE for edu- 
cation practice. The politics of quality bears a self-reinforcing dynamic which 
its actors largely share. Some critical rhetorical questions thus arise. If the lack 
of quality can be addressed only through more QAE, when is the promise of 
quality finally realised? Are we in an endless loop of planning an increasingly 
stringent, effective, all-encompassing system which will provide better evidence 
for decision-making? 


Authorising but diverted governance 


Due to the widespread view of how QAE education processes help to boost 
the quality of education and consequently the economy, the implementation of 
QAE confers much legitimacy. In policymaking, QAE data are used to legiti- 
mise decisions, a phenomenon extensively studied in comparative education 
and other research strands. In our study, instead of merely serving to legitimise, 
we found that transnational and national expert networks gained more room 
for action seeking justification with the help of QAE data. However, as in 
the previous dynamic, the implementation of governance seldom follows the 
original plan. 

Authorisation happens on different scales. Brazil, China, and Russia have 
gradually gained greater geopolitical prominence and self-confidence. This is 
manifested in their decreasing dependency on international funding since the 
1980s and 1990s, their growing roles in international organisations, and their 
increased transnational cooperation (Chapters 3 and 4). Geopolitical change 
was by no means due to QAE; rather, its new, more powerful role could also 
ensure more finely tuned governance, drawing on growing economic capacity 
and, arguably, more articulate global aims. 

Chapter 3 traced the backdrop to the newly shaped political situation and its 
facilitation and restriction of the creation of QAE systems between the mid- 
1980s and the mid-1990s. There was an opportune moment for the introduc- 
tion of QAE in each context. The construction of Brazilian QAE combined 
problematic school expansion, federalism, decentralisation, and democratisa- 
tion, at which point a QAE system seemed a feasible solution to govern a 
complex and politically delicate education system. Similarly, democratisation in 
the totally different post-Soviet context can be argued to have been a catalyst in 
Russia, but the system fragmented more dramatically because of marketisation, 
the devolution of central power, and education financing. This was an oppor- 
tune moment for international organisations to introduce their perspective. In 
China, economic growth was seen as contingent on education and there was a 
longer tradition of QAE. In each context, international organisations’ presence 
supported reforms in shaping the agenda. 

QAE offered an apt solution at the historically opportune moments of 
democratisation and the state’s crisis of legitimisation or its attempts to curb the 
effects of decentralisation. At such moments, international organisations and 
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cooperation helped to introduce QAE as a solution (Chapter 3). QAE was a 
key tool in steering this development and inserting general political aspirations 
into the education system. It is thus interesting to ask whether QAE has served 
the quality of education or the need to control. These two are not mutually 
exclusive, but the internal logic is that quality can only happen through tighter 
control. 

As we have already established, there were variations in the different coun- 
tries; reactions relate to the longer course of history in each context. In line 
with the transnational QAE agenda and drawing on the Soviet planning tradi- 
tion, the expert community in Russia treats the “science of testing” in positivist 
terms and sees it as a valuable method for understanding social phenomena 
(Piattoeva & Gurova 2018). Although Brazilian experts’ connections with inter- 
national actors have been clear for decades, neither the idea of imposition nor 
regional factors seem likely causes of variation based on our review but rather 
the local relearning of QAE techniques (Kauko et al. 2016): Brazilian education 
policy has adopted practices from the global QAE agenda but has reflected the 
underlying ideology (Minina et al. 2018). The Chinese case is more nuanced, 
as shown in Chapters 4 and 5: experts can find some room for action, but the 
state authorities monitor their action more closely. 

Various questions of authorisation arise in expert communities’ room for 
action. Chapters 4 and 5 point to how expert communities are well-networked 
transnationally at the same time as they negotiate the area of their operation. 
For example, Russian experts are supportive of the QAE technology of the 
international organisations conducting international comparative assessment 
studies, while gaining authority and legitimacy for themselves. At the same 
time, they explicitly attempt to link international studies to the national context 
(Piattoeva & Gurova 2018). Experts also express views rooted in domestic ped- 
agogical traditions, including QAE practices, indicating a wish to retain what 
they see as valuable in these traditions (Chapter 8). The grip of state authority 
on expert communities has somewhat weakened in China and Russia, while 
such control in Brazil has been almost absent. In all countries, the influence of 
expert communities has grown in ways policymakers have not always predicted. 
QAE’ prominence in the education agenda enables expert communities to 
strengthen, because experts are among the few actors capable of producing data, 
the core commodity for sustaining the QAE system. 

The implementation of QAE gets more diverted than meets the decision- 
maker's eye. Criticism of the theoretical notion of governance at a distance — 
the replacement of formal prescription with subjects’ “voluntary” commitment 
to accountability through performance evaluation — was not among our initial 
premises. However, in the course of our research, it became increasingly obvi- 
ous that this notion, typically formulated with reference to education policy 
developments in European countries, is a flawed perspective for an analysis of 
QAE as a model of governance in our case countries (which may or may not 
apply in other contexts). Our analysis has shown that as the interests of different 
groups of actors in how they react to QAE tools diverge or even collide, the 
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aim of governance at a distance is only partly achieved in education practice — 
or sometimes not at all. 


Destabilising and reorganised role-setting 


Our study found that QAE created new roles and changed old ones in differ- 
ent scales of operation. One of the key dynamics in the politics of quality im 
education thus lies in QAE’s contribution to both destabilising and buttressing 
actor roles. 

Chapter 4 sought an understanding of actor constellations in national politi- 
cal arenas and their connection with changes in the QAE infrastructure. Dif- 
ferences in this reorganisation emerged between the Brazilian case, where a large 
body of third-sector organisations had mobilised, and the less dynamic arenas 
of China and Russia. QAE has the potential to liberalise and marketise educa- 
tion because feedback channels from the education system are open and the 
formation of new civic actors proliferates to provoke debate and spark new 
movements. As Chapter 5 pointed out, this is clearly not the case in China and 
Russia. However, as Chapter 4 demonstrated, even in China and Russia, we 
could observe the expert communities” growing transnational networking. In 
relation to these countries” rising geopolitical aspirations, there is a tendency 
to curtail the significance of international organisations at the national level. 
These countries have been able to strengthen their steering of international 
organisations on quality in national education, while new actor constellations 
have created new degrees of instability, even if such instability is less marked in 
China and Russia. 

As we concluded in Chapter 5, 


Policymakers’ and experts’ basic relationship with assessment data use 
differs. Whereas policymakers can work with or without data, experts 
depend on it. Whereas policymakers can bend interpretations, experts 
attempt to adhere to what is analysed. Experts’ independence from state 
organs is another important issue. These differences in understanding data 
use also reflect the basic dynamic of the relationship between the state 
and experts. 


The destabilising potential of politicking may be seen in QAE’s influence on the 
latent conflicts between state and experts. Experts do not aspire to the role of 
political decision-makers, but they may be uneasy about how policymakers use 
data. However, policymakers understand the power of QAE data and the need 
to control its use (see Chapter 5). 

In Chapter 7, our analysis of the effects of QAE on relations between local 
authorities and schools added a combination of governance theories and organ- 
isational analysis to our theoretical perspectives, to provide a better understand- 
ing of how QAE mechanisms could both provide and limit schools’ access 
to new sources of power. While the comparative analysis of local education 
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practitioners added to the critique of the simplistic notion of governance at a 
distance, it went further by uncovering the diverse ways in which QAE policies 
can enhance and change local governance and outlined the potential room for 
action of schools subjected to performance evaluation. This room for action 
was more complex than “adaptation” or “resistance”: some consequences and 
results had no connection with the original purpose. 

It is clear the data being collected also have consequences for schools’ posi- 
tion in a societal setting so complex that school-level actors see them not only 
as evidence for decision-making but as a game in which their own working 
conditions or moral choices are at stake. In this sense, QAE works according 
to a dynamic in which the schools seek to perform to governance, govern- 
ance seeks to perform to policymakers, and policymakers seek to perform to 
their international and national peers and the public. This may also be seen 
as a regular accountability chain. However, we have already argued that QAE 
power at the local level eventually changes actor relations, albeit not uniformly, 
as Dahler-Larsen (2012) predicted. This, and the fact that all data flows include 
friction and productive power (Chapter 6), raises the question of QAE’s actual 
efficacy as a governance tool. 

As our research findings in Chapter 7 indicated, the local picture of the 
different actor roles in education is unsurprisingly complex. Russia is a good 
example of layeredness, whereby a mixture of different QAE models has pre- 
vailed during different periods. In the Russian case, ascribing change as con- 
vergence towards the West is therefore questionable. For example, as a Soviet 
legacy, equality of access remained a major principle in policy discourse and 
a legitimation of QAE (Gurova 2017). Nevertheless, surveillance techniques 
are being expanded in Russia with the help of equality discourse and merged 
with more traditional QAE practices (Piattoeva 2018). The Russian data sug- 
gest the local level suffers from the increased bureaucratisation of school work 
resulting from current education governance, which aims to account for every- 
thing (Gurova 2017). Russian teachers face a series of moral questions regarding 
the effects of QAE, in which policy and professional integrity are juxtaposed 
(Gurova & Piattoeva 2019). In the face of the multiple QAE techniques in use 
in Brazil (Kauko et al. 2016), schools may adopt different positions towards the 
intended control (Candido et al. submitted). In China, increased control has 
also increased the implementation responsibilities at the local level (Suominen 
et al. 2017). Although there are ways in which control is legitimised, there are 
serious degrees of destabilisation in local-level actors’ work. 

As researchers, we have had to come to terms with the destabilising poten- 
tial embedded in QAE. In Chapter 2, in examining our research, we noted 
that we could learn something of QAE through the research process itself. It 
seemed that our participants reacted to our QAE research much as they would 
to QAE procedures themselves, either seeing it as a token or refusing coopera- 
tion. Bureaucracy sometimes sucked us into Kafkaesque mazes, which afforded 
a glimpse of the everyday reality of many of our interviewees. However, we 
formed the impression, especially at the local level, that the voice of single 
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teachers was not often heard and that they found a moment of empowerment 
in discussing the QAE steering tools which affected their everyday lives. 


Behind and beyond quality 


In relation to these dynamics, we believe we have made a convincing argument 
that the complexity of understanding the phenomena of politics of quality is 
embedded in the attempt to understand how it enables or hinders action. We 
started with the idea that politics is a means to control contingency. Where 
quality is concerned, QAE is a means to attempt to control what is done at dif- 
ferent settings of action across all the contexts and institutions we have studied. 
We have elaborated this from the perspective of complexity, where it can be 
argued that attempts to control contingency increase the probability of action 
of one kind, while the probability of another decreases. Concretely, we argue 
in our research that the three dynamics described here are patterns which have 
become more probable in the politics of quality in education in Brazil, China, 
and Russia. 

We recognise some limitations in our argument, however. Limited access to 
interviewees and observations and the issue of sensitivity in our data collection 
described in Chapter 2 have implications for the reliability and validity of the 
findings we report. Where reliability is concerned, we can first say that with 
better access to desired data in China and some of the potential key informants 
at the national level in Russia, we might have produced a more comprehensive 
and more nuanced picture of actor relations, data circulation, and the QAE 
enacted at the local level. At the same time, it is important to note that the 
concern here is not with the generalisability of our findings in the sense of 
their quantitative representativeness — even with more interviews and obser- 
vations, our body of data in this sense would of course have remained very 
small. Instead, we emphasise that the value of our research’s findings should be 
assessed by enquiring the extent to which we have contributed to a qualita- 
tive understanding of the issues we have analysed, for example, of the range 
of views expressed by interviewees rather than the perception of “the typical 
view” among different groups of actors or in each of the case countries. 

With these reservations in mind, we certainly do not claim that our list of 
dynamics is exhaustive. As Chapter 2 discussed, there might be other possible 
interpretations of the research material we collected, because there might have 
been other foci. As this book concludes, we attempt to open some possible 
avenues for other interpretations, which may encourage further research on the 
politics of quality in education. 

The notion that QAE has overtaken quality as a goal of education policy 
raises an interesting general question about how education is steered. There 
are many studies on the different techniques of governance, but this idea might 
be tested and developed by a deeper investigation of the comparison of policy 
and practice. This would especially require research on the use of knowledge 
in decision-making at different levels, possibly drawing on observation data, to 
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enable an analysis of the extent to which QAE data are really used or to which 
they are a performance or fabrication. If data are not used, why gather them? 

The way in which governance becomes more complex while authorising 
new actors to function in the political arena is interesting. We have clearly 
indicated the endless bureaucracy involved in QAE, but during this project’s 
long journey, we have grown increasingly interested in the local level of action 
in the contexts we have studied. We continue the analysis based on our data 
and stress the importance in understanding thoroughly how local actor rela- 
tions change. 

The destabilising feature of QAE policies, which reorganises actor relations, 
is an interesting finding from the perspective of contingency. While quality pol- 
icies are a means to control contingency, in other respects, contingency grows. 
This is a particularly interesting finding in the cases of Russia and China, where 
increasing contingency is not always considered politically desirable. Neither 
seems to conform to the argument of the state’s diminishing role in the face of 
globalisation, which some of the researchers discussed in Chapter 1 expected. 

The emphasis in the governance-at-a-distance literature has been pre- 
dominantly a phenomenon of the Global North, with its theory having its 
roots in France and the UK. Our focus has been on large countries, two of 
which, China and Russia, are quite hierarchical, while the other, Brazil, is less 
top-down managed. None of the yielded results support the idea that the 
governance-at-a-distance theory is a good description of what we have observed. 
We do not claim to offer universal results with our data, but we believe it could 
well be opportune now for scholars outside these vast linguistic and territorial 
regions to take note of the serious doubt cast on this theoretical tradition. 
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The interviews are coded according to the interviewees’ positioning in the 
education arena at the time of the empirical work. However, most interviewees 
occupy or have occupied different positions throughout their careers, simul- 
taneously or not. Their action benefits from the knowledge and the networks 
built within other scopes and scales of action. To ensure we preserve the inter- 
viewees’ anonymity we will not share this detailed information. 


People acting as politicians or civil servants at People acting in or for: 

N = National level (federal or union) IGO = International governmental 

SN = Sub-national level (region in organisation 
Russia, state in Brazil, province in INGO = International non-governmental 
China) organisation 

M = Municipal level (municipal NNGO = National non-governmental 
government, city in China) organisation 

People acting as: SNGO = State non-governmental 

E = Expert (engaged as consultant, organisation 
lecturer, and so on) MNGO = Municipal non-governmental 

Events: organisation 

SM = Seminar PO = Private organisation 


S = School (teachers, principals, coordinators, 
psychologists, social pedagogues, students) 

MS = Municipal school 

MBS = Minban school (non-state operated 
school) 

PS = Public school 

SS = State school 

MD = Media 
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Coding of interviews 


Analysis tools 


To ensure a valid and equivalent analysis of the interview material, the Chinese 
team at the University of Turku used NVIVO 11, whereas the Brazilian and 
Russian teams at the University of Tampere used ATLAS.TI 7. While both 
were equally adequate for the analysis we needed, the final choice depended on 
practical considerations, such as the availability of the programme licence and 
the home institution. Since both programmes allow input in multiple languages 
and the researchers working with the coding were fluent in both languages, 
we did not translate the material from Chinese, Russian, and Portuguese into 
English but simply did the coding in English. Translations were later done for 
the coding units used as citations in the chapters. 

Coding units (ATLAS.TI: quotations, NVIVO: references) refer to 
the block of material selected for coding. The rule for delimiting the cod- 
ing unit was the topic/issue captured within a piece of text, rather than the 
physical linguistic units (e.g., word, sentence, or paragraph). To make them 
fully understandable, selected coding units included necessary contextualisation 
information (material before and after). Sometimes this required enclosure of 
the interviewer's questions or large sections of the text in which the inter- 
viewee digressed but then returned to the topic, because if this were considered 
as two quotations instead of one, the second would make no sense. 

It must also be noted that coding units were not mutually exclusive, and the 
same coding unit might belong to more than one category. For example, in the 
Chinese case, interviewees frequently mentioned the challenges for education 
and the existing unequal situations in Chinese basic education together. There- 
fore, these coding units were coded under the categories of “challenges” and 
“inequality”, as they revealed relevant information about both topics. 

The interviews were exhaustively coded. The exhaustiveness of coding 
implied that even when the issue was not directly connected with the main 
research question (e.g., about childhood education), this part of the text was 
nevertheless coded. 

The category definition (ATLAS.TI: codes/sub-codes; NVIVO: notes/ 
sub-notes) includes the category name (ATLAS.TI: code name; NVIVO: 
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name) and its description (ATLAS.TI: comment; NVIVO: description). The 
latter includes what is meant by that sub-category or category, which features 
are included, the criteria or decision rules, and other relevant information for 
understanding the coding's structure. In brief, categories” names and descrip- 
tions make a coding frame which allows the coder to code the material and 
“outsiders” to make sense of the coding. 

The main categories refer to what the researcher wishes to know within the 
research, whereas sub-categories normally denote the substance of the main 
codes. They are therefore frequently data driven. Categories or sub-catego- 
ries describe the selected coding units (Qualitative Content Analysis: coding 
frame, coding manual, or code book; ATLAS.TI: coding list; NVIVO: node 
list). Whereas the same coding unit may belong to more than one category, the 
main categories should cover only one aspect or theme of the material, and 
sub-codes within a given code should be mutually exclusive: there cannot be 
two or more referring to the same topic within that main category. 

For example, in the coding of the Brazilian interviews, we had the following 
coding unit: “The IDEB [Development Index of Basic Education], it works 
with targets, it is in that framework of the PDE [Education Development Plan] 
of 2017, and there was an effort of the Government in finding some type 
of meeting point between the scale of Programme for International Student 
Assessment (PISA) and the scale of the SAEB [Evaluation System of Basic Edu- 
cation on which the IDEB is based].” Although this segment of text could have 
been cut into smaller coding units, we chose to keep it in its entirety to preserve 
its full meaning and context. However, since it covered several relevant issues, 
we coded it with six different sub-categories belonging to four categories, 
namely: “Quality evaluation: national instruments: IDEB”; “Quality evalua- 
tion: national instruments: SAEB”; “Quality evaluation: international instru- 
ments: PISA”; “Quality evaluation/improvement: paths of influence”; “Quality 
improvement: Federal Government Programme: PDE”; and “Actors: relations: 
INEP-Federal Government”. The first category, “quality evaluation”, is divided 
into several sub-categories, among which some refer to the national and inter- 
national instruments used to assess quality. In turn, to analyse the material as 
accurately as possible, we have created sub-sub-categories with the names of the 
instruments mentioned (IDEB, PISA, and SAEB). It is also clear in this segment 
of text that visions about how quality evaluation might be improved (category 
“quality evaluation/improvement”) were influenced by concrete develop- 
ments, practices, or policies. This was frequently detected and coded under 
the sub-category “paths of influence”. In this case, the influence was clearly 
PISA. This sub-category could indeed have been further divided, for example, 
by describing the exact influences, as was previously done for “inter/national 
instruments”. However, since the number of coding units was much smaller 
and simultaneously more homogeneous (most mentions referred to the same 
influences), this would have led to some imbalanced sub-categories, as some 
would have one or two coding units, while others would gather almost all the 
coding units of that sub-category. We therefore decided to keep them all in one 
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sub-category. This coding unit also refers to two other topics. One was a federal 
programme which was perceived as a tool for quality improvement, and which 
was in turn connected with national and international instruments of quality 
evaluation. This was thus of major relevance. It was then coded under a sub- 
category which referred to federal programmes formulated to improve educa- 
tion quality. The programme itself provided the name of the sub-subcategory 
according to the logic described for instruments. Another topic implicit in this 
coding unit was the relationship between the agency in charge of the assess- 
ments, the INEP (National Institute for Educational Studies and Research) and 
the government. The INEP designed the SAEB and IDEB, but the effort of 
making them compatible with PISA was not only technical for the INEP, but 
also a clear political effort from the government. This reveals something about 
the relationship between the INEP and the government, and it is therefore 
also coded under the sub-category referring to relationships between actors, 
particularly the relationship between the INEP and the federal government. 


Validity and coordination within and between teams 


Using the interview guidelines (Appendix 3), we discussed the initial set of 
categories, which mainly covered areas that we wanted to explore further, and 
this initial coding frame was shared between all three teams but subsequently 
adjusted to fit each country case. All teams had several persons engaged in cod- 
ing, allowing crosschecking at different stages of the process. The coding frame 
was constantly refined and updated during the coding process. At the end of 
each working package, researchers usually revised and expanded the categories. 
In brief, within each team, all involved in coding discussed changes in the cod- 
ing frame. The categories were not readapted or forced on new pieces of data. 
Instead, if categories did not completely match data, the differences or particu- 
larities were reported in the category’s description, or a new sub-category was 
created to fit the material. 

For example, in the Russian case, this refinement process led to the crea- 
tion of new sub-categories at the end of the coding process. There were thus 
several sub-categories which emerged as a result of later analysis of the whole 
body of data. A concrete example is the sub-category “visibility”, which signi- 
fies cases when data collections were justified by respondents to demonstrate 
(to the public and officials) evidence of good or bad performance, locating 
certain problems, and so on. We saw the relevance of this sub-category only 
when we had reviewed the whole coding. The frequent revision of the coding 
frame and the exchange between researchers were fundamental to the coding 
of interviews. 
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Interview guidelines 


Main idea 


The main idea behind this guideline is to set out common core questions 
essential for the goals of each of the four main themes. The main goals of each 
theme and the core questions offer a basis for balancing the questionnaire and 
adjusting it to country-specific needs. 


Methodological notes 


Prior to the interview, the interviewer should have a good understanding of the 
role of that organisation (school, government, international organisation, non- 
governmental organisation, and so on), based on documents and a literature 
review. Depending on the time frame and nature of the interview, it is also pos- 
sible for the interviewer to allow the interviewee briefly to present himself or 
herself to gain an understanding of his or her position in the organisation and 
his or her role in QAE (Quality Assurance and Evaluation) dynamics, as well as 
the organisation's position and role in such dynamics. 


Terminology 


CC = common core question, a question presented in all three BCR (Brazil, 
China, and Russia) cases 

CFU = common follow-up question, a question presented in all three BCR 
cases, however adapted or omitted, depending on the answer given to CC 

LS = level-specific question, a question to be asked only on a specific 
level (transnational, international, federal, national, state, regional, local, 
school) 

AS = actor-specific question, a question to be asked only of a specific actor 
group (for example, teacher or politician) 

IOs = international organisations (governmental and non-governmental, 
i.e., IGOs and INGOs) 

LO = local organisations (e.g., NGOs) 

Actor = adjust terminology according to the person being interviewed 
(i.e., refer to organisations, people, or actors generally) 


Theme 1: Introduction 
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First goal of the theme: to understand the respondent’s view on quality and evaluation 


in basic education. 


Priority Questions 


Notes 


i CC How do you understand 
“quality in basic education”? 

1.1 CFU Should quality be assessed? 
Why? 

1.2 CFU How is quality assessed in 


Brazilian/Chinese/Russian 
basic education? Is there 

a national/regional/local 
quality/evaluation system? 
Please give concrete 
examples. 


1.3 LS/AS Is there a difference between 
“quality of education” and 
an “education of quality”? 


2 CC How do you understand 
“evaluation”? 


2.1 CFU How do you see the 
relationship between 
“quality” and “evaluation”? 
3 CC From your perspective, what 
are the most important 
problems and challenges in 
Brazilian/Chinese/Russian 
basic education? Why? 


Order of questions 1.2 and 1.3 can be 
changed — adjust according to the answers. 

Direct connection may be made with 
testing regimes when appropriate; decide 
according to answers. 

Questions about quality and/or assessment 
practices and policies’ implementation may 
be asked. Adjust accordingly: Which actors 
have participated in the process? What 
were their roles in the process? Who has 
provided support/expertise/funding? 

If more than one quality/evaluation system is 
mentioned, it may be asked why they were 
implemented if there were already other 
main quality and/or assessment practices 
and policies. 

Questions may be asked about how the main 
(and other) quality and/or assessment 
practice and policy have been received by 
public opinion and other stakeholders. 

May be more relevant to Brazilian and 
Chinese cases. Adjust according to answer 
to question 1. 

The idea is to understand and explore the 
eventual differences — if any — between 
“Quality of Education”/“Education 
Quality” and “Education of Quality”/ 
“Quality Education” /“Education Quality”. 


Adjust according to answers to question 
1, perhaps as a follow-up question, but 
this may be unnecessary because the 
interviewee's answer included it already. 

As above. 


Adjust according to the country in question. 
For IOs/Los, perhaps omit country or 
understand if they are talking about their 
particular country or in general (to make 
an explicit differentiation). 


(Continued) 
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Priority Questions Notes 


3.1 CFU Are there differences in Adjust according to answer to question 3. 
opinion between the various This may have already been mentioned. 
actors involved? Please note this answer may be more 

related to the following theme (actors), 
depending on the detail into which the 
interviewee goes at this point. 

3.2 CFU Have the problems/challenges/ Adjust according to answer to questions 3 
opinions changed recently? and 3.2. This may have been mentioned 
When? already. Please note that this answer may be 

more related to dynamics, depending on 
the detail into which the interviewee goes 
at this point. 


Second goal of the theme: to understand the interviewee's concrete involvement in — and 
its perceived impact on — QAE policy and practice. (Only used if it is impossible to gain 
information beforehand.) 


Priority Questions Notes 
3.3 CFU How did you come to work This serves as the main question for the 
with basic education/ second goal of theme 1 but is optional. 
evaluation/ quality? 
3.4 LS/AS What is your role/the role Adjust according to interviewee and previous 
of your organisation/unit/ answers. 
department? 


3.5 LS/AS Do you (or your organisation) As above. 
produce or use any data 
related to quality and 
evaluation in education? 


3.6 LS/AS Are you indirectly/ As above. Main idea: to understand if 
directly involved in they have — or perceive that they 
national/regional/local have — an impact and, if so, what kind 
policymaking? (e.g., discursive, pragmatic)? 

3.7 LS/AS What is your role/the role Adjust according to interviewee and to 
of your organisation/unit/ answers to question 1.3/2.3 and 3/3.1. 


department in seeking 
to address the problems 
[mentioned in question 3]? 
3.8 LS/AS What are the main challenges Adjust according to actor and level. 
in your own work/action? 


Theme 2: Actors 


First goal of the theme: to identify the main actors (collective or individual) in the field 
and their role/action and perceived impact in quality and evaluation policy and practice. 
(The first goal concerns only macro- and meso-level actors.) 
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Priority Questions Notes 

4 CC Which actors would Adjust question concerning the level (human 
you consider the most or organisation actors; education field or 
important in shaping basic community; passive or active approaches; 
education policy? Please and so on. Perhaps use the language of the 
describe who they are. interviewee so terminology is not an obstacle). 

4.1 CFU Does their importance vary Order of questions 4.1, 4.2, and 4.3 may be 
according to policy issue? changed — adjust according to the answer. 

4.2 CFU How do these actors deal Adjust accordingly. 
with the problems you 
identified previously [in 
answers to questions in 
Theme 1 and to 4.1]? 

4.3 CFU What has been the role of Perhaps already answered. Adjust accordingly. 
standardised testing in Main idea: to understand if the performance 
this context? In which evaluation/indicators/rankings and so on 
measure and by whom? are part of the solution and how, why, and 

by whom (in the eyes of the interviewee). 

4.4 LS/AS What sources of Adjust according to level and actor. 
information do they use 
for evaluating quality [in 
answer to question 4.3. 
or previously]? 

4.5 LS/AS Are these data reliable? Adjust according to level and actor. 

Please explain. 

4.6 LS/AS Who uses this information Adjust according to level and actor. 
and for what? Please give 
examples. 

4.7 Optional: Are international organisations Adjust according to previous answers and level/ 

CFU or experts involved in the actor. Perhaps already mentioned. Main idea: 
LS/AS practices you mentioned [in to understand the role of international experts 


4.2. and 4.3]? 


and/or organisations. 


Second goal of the theme: to understand the position of the interviewed actor in the field, 
as well as the connections/relations between different actors. 


Priority Questions Notes 
5 CC Who are your main partners in Adjust accordingly: basic 
dealing with basic education? education policy/local 
community/and so on. 
5.1 CFU, With which other actors do you 
LS/AS (and/or your organisation) interact 
regarding basic education issues? 
5.2 CFU In which projects/programmes do 


you cooperate and how? Please give 


examples. 


(Continued) 
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Priority Questions Notes 

5.3 CFU Why are you not working with all the Main idea: to investigate if 

actors [in answer to question 4]? there are any reasons for not 
working with some of the 
actors mentioned above. 

5.4 CFU, Are you/your organisation/ Adjust accordingly. Perhaps 

LS/AS department involved in quality redundant, depending 
and/or evaluation practices in on the previous answers 
[Brazil/China/Russia OR in this (especially to 3.4). 
region/state/municipality]? 

5.5 LS/AS Are you personally involved in these Several follow-up questions 
developments? In what capacity possible, adjust to level/ 
and since when? For how long have actor and previous answers. 
you been (and will you be staying?) 
in this position? Where and what 
was your previous position? 

5.6 CFU Can you establish a time line Possibly serves as a bridge to 


regarding the roles of the actors [in 
answers to questions to Theme 2]? 
Have their roles changed? How 
and why? 


Theme 3: Change dynamics 


the following theme. Adjust 
accordingly. 


First goal of the theme: to understand changes in actor relations. 


Priority 


Questions Notes 


6 CC Why did you start working The question may be asked negatively: 


with the partners you 
mentioned previously [in 
question 5]? Why then? 


why have you never worked with these 
actors [not mentioned in question 
5; name actors unmentioned by the 


interviewee or those unmentioned 


as partners]? OR Why have you 
discontinued working with the partners 
you mentioned before [in question 5]? 
6.1 CFU How would you describe Perhaps answered before. Adjust 
your relationship with accordingly: competitive, cooperative, 
the partners with whom coalitional, conflicting, and so on/ 
you work [mentioned in voluntary, mandatory, negotiated, and 
questions 5 and 6]? so on. 
6.2 CFU, How would you describe Adjust according to level and actor. 
LS/AS the relationships between 


partners in the field 
of basic education in 
general? 
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Second goal of the theme: to understand the role “quality” and “evaluation” practice and 
policy have played in the changes of the first goal of this theme and to identify whether 
and how they define relations among and between actors (or vice versa). 


Priority Questions Notes 

8 CC How does your work with these May be asked in the previous 
actors [mentioned in question theme [question 5]. Depending 
6] help to tackle the problems on answer, may also be asked 
in basic education mentioned negatively: how do relations 
earlier [in Theme 1]? among actors impede the 

tackling of problems? 

8.1 CFU, In what respect are these relations Perhaps already answered. Adjust 

LS/AS consensual? Are there differences accordingly. 
in opinions/practices/goals 
between different actors in 
tackling the problems in basic 
education mentioned earlier [in 
Theme 1]? Why? 

9 CC Have there been changes in Adjust accordingly for content, 
dealing with basic education division of labour between 
problems? Why? actors, share of responsibilities 

between different levels, and 
so on. 

Depending on answer, may also be 
asked negatively: why didn’t this 
change? 

9.1 CFU Who pushes for changes? What 
are the main reasons? Who 
resists these changes? Why? 

9.2 CFU What has initiated or been the Perhaps already answered. Adjust 
reason for these changes? accordingly. 

9.3 CFU Which actors were behind these Perhaps already answered. Adjust 
changes? accordingly. 

9.4 CFU, Do you think that the (recent?) Perhaps already answered. Adjust 

LS/AS developments in quality and according to previous answers, 
evaluation are involved in these level, and actor. 
changes? In which measure? 
Please explain. 

9.5 CFU, Are these changes peculiar to Adjust accordingly. Main idea of 

LS/AS Brazilian/Chinese/Russian this question: to understand the 
OR to this state’s/region’s/ horizon of interpretation. 
municipality’s education? 

9.6 CFU What are the benefits/negative Perhaps already answered. Adjust 


effects of these changes? How 
well do you think Brazil/ 
China/Russia has accepted such 
changes and adapted to them? 


accordingly. 
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Third goal of the theme: to understand transnational connections and their impact on 
national and sub-national policy and practice. 


Priority Question Notes 
10 CC, Have Brazilian/Chinese/Russian Questions may be asked 
LS/AS policymakers/teachers/and so on used the about particular 
experience of policy /practice from other programmes, actors, or 
local/regional/national/international policies. 


actors? From where, why, and how have 
they used such experience? 


11 CC Do you think that Brazilian/Chinese/ Questions may be asked 
Russian developments in evaluation about particular 
and/or quality practice and policy have programmes, actors, or 
inspired (or been used by) others locally/ policies according to 
regionally/nationally/internationally? context. 


Which, how, and why? 


Theme 4: Future 


Main goal of the theme: to understand the actors’ expectations and the possibilities they perceive. 


Priority Question Notes 


12 CC How do you see the future trajectories in 
basic education in the short and long term? 
12.1 CFU Are these related to quality and evaluation in Perhaps already 
education? answered before. 
Adjust accordingly. 
12.2 LS/AS Which actors do you think will be involved in 
the trajectories? How and why? 
12.3 LS/AS How do you see your work/institution's 
activities involved in these developments? 
Questions may be asked 
about particular 


developments. 
13 CC In an ideal world, what would be the perfect 
trajectory [considering what was mentioned 
in answer to question 3]? 
13.1 CFU What changes do you consider necessary Adjust according to 
in basic education? Would you change interactions, data, 
anything related to evaluation and/or actors, and so on. 


quality policy and practice? 
13.2 CFU How would your work (institution) 
contribute to this perfect trajectory? 
Questions may be asked 
about particular 
developments. 


Index 


Page numbers in italics indicate figures and in bold indicate tables. 


abductive research design 19 

academic research in China 102—105 

accountability of authorities 150—151 

action see room for action; use of political 
space in CADEP framework 

actors: in Brazil 80—82; in China 82-84; 
destabilising and reorganised roles of 
186—188; in education 1;in governance 
at a distance 117; interactions with 
agendas 71—73, 79—87; in Russia 
84-87; schools as 140—141; states as 7; 
transnational expert communities as 
87—88; in transnational network 5—6; 
trust and mistrust between 132—133; see 
also expertise/experts; political situation 
in CADEP framework 

agendas: as changing, in 1990s 73-75; 


interactions with actors 71—73, 79-87; of 


international organisations 182—184 

archetypal images of future 163—164, 
175-176 

assessment system: in China 54-56, 77-78, 
82; in Russia 56-57, 85 

audit society 94-95 


borrowing and lending theories 4 

Brazil: archetypal images of future in 
175; building capacity in 75-77; civic 
movements in 80—82; data circulation 
in 118—119; data collection in 27-28; 
historical trajectories and pushes of 
present in 164—166; path towards QAE 
in 45-51,61; perceptions of future in 
166—168; political instability in 33-34; 
rise of school performance evaluation 
in 142—143; Santa Catarina 141; school 


visits in 38; state and expertise in 
99—101; see also Evaluation System of 
Basic Education 

BRICS 7-8, 63-64; see also Brazil; China; 
Russia 

bureaucratic-professional model of 
governance 138 


CADEP framework see Comparative 
Analytics of Dynamics in Education 
Politics (CADEP) framework 

case countries 7—9; see also Brazil; China; 
Russia 

centralisation of education in Russia 56—57, 
60 

Cheboksary, Russia 141—142 

China: actors in 82—84; archetypal images 
of future in 175; building capacity in 
77-78; data circulation in 119; data 
collection in 28-29; engagement 
with international community 73-74; 
historical trajectories and pushes of 
present in 168—169; official approval 
in 34; path towards QAE in 51-56, 61; 
perceptions of future in 169—171; rise 
of school performance evaluation in 
143-144; school visits in 38-39; state and 
expertise in 101—105; see also National 
Assessment of Education Quality 

“China Dream” 169 

circulation of data: in Brazil, China, and 
Russia 118—121; data availability and 
123-127; data production and 121—123; 
data use and 127—131; governance at a 
distance through 116—118, 131—133 


civic movements in Brazil 80-82 


206 Index 


Comparative Analytics of Dynamics in 
Education Politics (CADEP) framework: 
conduct of research within 39-41; 
dimensions of 9, 21, 40; overview 8, 21; 
paradox in 181; political possibilities 
in 22-23; political situation in 21-22; 
project consortium and work mode 
11-12; reflective approach of 18-19; 
systematics of research procedure 25-32, 
26; use of political space in 23—25; see 
also dynamics in politics of quality 

comparative education and policy transfer 
4-5 

comparing dynamics method/perspective 
19-21 

complexity theory 3,9, 161, 181 

contingency 3, 45 

critical futures studies 161 


data and data systems: in Brazil 100—101; 
in case countries 110-111; in China 
102-105; expertise and 97—98; in 
governance at a distance 115—118, 
131-133; in Russia 106—109; see also 
circulation of data 

data collection: for project 27-31, 141—142; 
reliability of 132; in Russia 85-86; by 
states 92 

decentralisation of education: in Brazil 
45-46, 49-50, 118; in China 51-52, 55; 
in Russia 58 

destatization 97 

discourse see political possibilities in 
CADEP framework 

dualism in domestic policy in Russia 
172-173 

dynamics in education: in case countries 
7-8; research focus on 19-21 

dynamics in politics of quality: authorising 
but diverted governance 181, 184—186; 
destabilising and reorganised role- 
setting 181—182, 186—188; overview 11, 
181—182, 182; shared and self-reinforcing 
goal setting 181, 182-184 


economic growth and learning 
achievement 6, 59 

education governance by data 115—116 

evaluation: defined 1; as internal 
management tool for schools 152—153; 
of local authorities 150—151; of schools, 
as indirect intervention 145—147; of 
schools, reliability of results of 156; see 
also quality assurance and evaluation 


Evaluation System of Basic Education (SAEB, 
Brazil): data availability and 126; data 
circulation and 118—119; data production 
and 122—123; data use and 127—128; 
overview 33, 49, 50, 116, 142-143 

expert community: influence of, in 1990s 
73; membership in 80; room for action of 
185; state and 109—110; as transnational 
87-88 

expertise/experts: capacity of anticipation 
and 91-92: context of 97-99: in 
production and interpretation of 
knowledge 91; roles of and relationship 
between state and 92-93, 109—111; state 
and, in Brazil 99—101; state and, in China 
101—105; state and, in Russia 105—109; 
see also expert community 


fieldwork: coping with politics and 
bureaucracy 33-36; manifestations of 
hierarchies and effects of QAE 36-39; 
self-reflection on 32-33 

framework, defined 8 

friction in data circulation 133 

future of QAE: concepts of time and 
archetypes of 162—164; overview 
161—162, 174—176; perceptions of, in 
Brazil 164—168; perceptions of, in China 
168—171; perceptions of, in Russia 
171-174 

futures triangle 163—164 


garbage can model 23-24 

global education industry 96 

global inspectorate or audit society 94-95 

globalism, as archetype of future 164 

governance: as authorising but diverted 
181, 184—186, 189; in China 55; as 
continuum 95; data as currency of 98; at 
a distance 115—118, 131—133, 139-140, 
185—186; local, QAE in 138—140, 
145—151, 157—158; national education 
systems and 95—97; networked 98; QAE 
as tool of 2; use of term 93—94 

governance technology 94 


historical paths towards quality assurance 
and evaluation (QAFE): in Brazil 45-51, 
61;in China 51—56, 61; commonalities 
and differences in 61—63; in Russia 
56—60, 62 

historical trajectories: in Brazil 164—166; in 
China 168—169; in futures research 163; 
in Russia 171—173 


“improvement through evaluation” 
discourse 145—147 

Index for Development in Basic Education 
(IDEB, Brazil) 99, 142—143, 166 

INEP (National Institute for Educational 
Studies and Research, Brazil) 99—101, 
118-119,123,126 

influence of schools 154—155 

INGOs in China 77,78,82-83 

inspections of schools, reward and 
punishment based on 147—150 

intended political project of data 115, 
131-133 

international, defined 2 

international influence and cooperation 62, 
63-64, 72 

international organisations (IGOs): 
agendas of 182—184; cooperation 
with, and building capacity 75-79; 
engagement with 73-75, 79; as 
epistemic communities 5; policies of 5, 
6-7; significance of, at national level 
186 

interpretation of research 19-21 

interview guide 25, 198—204 


knowledge production and expertise 91 


legitimation, defined 4 
liquid modernity 93 


marketisation of education: in China 
52—53, 54, 61; in Russia 59, 62 

material benefits of QAE policies 140—141, 
152-156 

meritocracy in China 101—102 

Ministry of Education: Brazil 47, 48, 
99-100; China 28-29, 52, 53, 55, 83, 
105; Russia 29-30, 35, 59, 86 

model, defined 8 


National Agency for Control and 
Supervision in Education (Rosobrnadzor, 
Russia) 29-30, 106, 108, 120, 121 

National Assessment of Education Quality 
(NAEQ, China): data availability and 
123—125; data circulation and 119; 
data production and 122; data use and 
128—130; overview 102—103, 116, 
143-144; perceptions of 171 

National Education Plan 2014-2024 (PNE, 
Brazil) 166, 167 

national education systems and governance 


95-97 


Index 207 


National Institute for Educational Studies 
and Research (INEP, Brazil) 99—101, 
118-119,123,126 

national large-scale assessment (NLSA) 121; 
see also circulation of data; Evaluation 
System of Basic Education; National 
Assessment of Education Quality; 
Unified State Exam 

national QAE policy communities 72—79 

networked governance 98 

new public management 139, 157 


organisational analysis 140 

Organisation for Economic Co-operation 
and Development (OECD): policies of 
5, 6; recommendations of 183; role of 
7; Teaching and Learning International 
Survey 77-78, 125; see also PISA survey 

Outline of National Plan for Medium 
and Long-term Education Reform and 
Development (China) 170 


path dependencies 4, 45, 162 

PIRLS (Progress in International Reading 
Literacy) 58 

PISA survey (Programme for International 
Student Assessment) 7, 74, 141 

policymaking: expertise and 91; ontological 
assumptions about 3—4; QAE as tool of 
2; ruptures of status quo and 71—72; see 
also expertise/experts 

policy monopolies 71—72 

policy process theories 21-22 

policyscape 24 

policy transfer: comparative education 
and 4-5; problems with idea of 181; in 
Russia 87 

political possibilities in CADEP framework: 
dynamics of 182; overview 21, 22—23, 
121 

political situation in CADEP framework: 
dynamics of 182; overview 21, 21—22 

political space see use of political space in 
CADEP framework 

politicking 182, 182, 186 

politics, defined 3 

politics of quality, as enabling or hindering 
action 188—189 

power: of measurements 117—118; QAE as 
tool of 2; of top-performing schools 155 

problematisation 23 

Programme for International Student 
Assessment (PISA survey) 7, 74, 141 

progress, constant, belief in 163 


208 Index 


Progress in International Reading Literacy 
(PIRLS) 58 
punishments for poor performance 


147—150 


Qualitative Content Analysis 31—32, 196 

quality assurance and evaluation (QAE): 
defined 1; as framing factor in education 
180-181, 182—184, 188—189; interest 
in 44, 61; in local governance 138—140; 
local governance of schools through 
145-151, 157—158; as political 18; 
resources gained through 140—141; 
schools’ room for action in 152—156; as 
tool to exercise power 2; transnational 
agenda of 5-7; see also dynamics in 
politics of quality; future of QAE; 
historical paths towards quality assurance 
and evaluation 


reactive sequences 45 

reflective research 18-19, 32—39 

reputation of schools 154—155, 158 

resource distribution to schools based on 
performance 153—154, 156, 158 

rewards for good performance 148—150 

role-setting, destabilising and reorganised 
186—188, 189 

room for action: in case countries 88; of 
expert communities 185; of research 
39-41; of schools 137—138, 152—158 

Rosobrnadzor (National Agency for Control 
and Supervision in Education, Russia) 
29-30, 106, 108, 120, 121 

Russia: archetypal images of future in 
175-176; building capacity in 78-79; 
Cheboksary 141—142; data circulation 
in 119—120; data collection in 29-31; 
engagement with international 
community 74; historical trajectories 
and pushes of present in 171—173; path 
towards QAE in 56-60, 62; perceptions 
of future in 173—174; political situation 
in 34-36; rise of school performance 
evaluation in 144—145; school visits in 
36-38; state and expertise in 105—109; 
state-dependent QAE in 84-87; see also 
Unified State Exam 


SAEB see Evaluation System of Basic 
Education 


scholars and policy in China 83-84 


schools: as actors 140—141; local governance 
of, through QAE 145-151; policy and 
137—138; room for action in QAE of 
152-158 

social construction of space 24 

Stalinist school model 56 

standardisation, defined 4 

state: changing role of 93-97; data 
collection and 92; as education actor 7; 
expertise and, in Brazil 99—101; expertise 
and, in China 101—105; expertise and, in 
Russia 105—109; relationship between 
expertise and 92—93, 109-111 

State Final Attestation (SFA, Russia) 
144-145, 173-174 

State Programme for Education 
Development, 2013-2020 (Russia) 173 

steering tools of governance 94-95 

supervision system in China 51—52, 53, 143 

symbolic benefits of QAE policies 
140-141, 152-156 

systematics of research procedure: data 
collection 27-31; overview 25-26; 
qualitative content analysis 31-32; themes 
of goals of data collection and analysis 26 


theory, defined 8 

TIMSS (Trends in International 
Mathematics and Science) 37,58 

transitional stages in Brazil 164—165 

transnational, defined 2, 20 

transnational agenda of quality assurance 
and evaluation 5—7 

“Transnational Dynamics of Quality 
Assurance and Evaluation Policies in 
Brazil, China, and Russia” 2 

transnational expert community 87—88 

transnational nature of work of experts 98 


Unified State Exam (USE, Russia): data 
availability and 125—126; data circulation 
and 119—120; data production and 
121-122; data use and 130—131; foreign 
expertise and 86-87; overview 59-60, 
116, 144—145; perceptions of 173—174 

United Nations Educational, Scientific and 
Cultural Organisation (UNESCO) 5, 6 

use of political space in CADEP framework 
21, 23-25 


World Bank 5, 6-7, 183 
world culture theory 4 


